Image coding method and image decoding method

ABSTRACT

An image coding method for coding an input image per block to generate a coded image signal includes: predicting for each prediction unit which is an area obtained by partitioning a target block to generate a prediction image of the target block; comparing a transform unit which is an area obtained by partitioning the target block and is a processing unit for frequency transform with the prediction unit, to detect part of a boundary of the prediction unit, the boundary being located within the transform unit; performing boundary filtering on the detected part of the boundary in the generated prediction image; calculating a difference between a filtered prediction image and the input image to generate a difference image of the target block; and performing frequency transform on the difference image for each transform unit.

TECHNICAL FIELD

The present invention relates to an image coding method for compressingand coding a moving image signal by performing frequency transform andquantization on a difference signal between an input image and aprediction image, and an image decoding method for decoding the movingpicture signal that has been compressed and coded in such a manner.

BACKGROUND ART

In the conventional image coding standards represented by InternationalTelecommunication Union Telecommunication Standardization Sector (ITU-T)standards denoted with H. 26x and International Organization forStandardization/International Electrotechnical Commission (ISO/IEC)standards denoted with MPEG-x, a picture is partitioned intopredetermined units and coding is performed on each of the partitionedunits. For example, in the H. 264/MPEG-4 AVC standard (for example,refer to Non Patent Literature 1), a screen (picture) is processed by aunit of horizontal 16 pixels and vertical 16 pixels, which is referredto as a macroblock. When motion compensation is performed, data can becompressed after a macroblock is partitioned into blocks (each of theblocks is composed of a minimum unit of horizontal 4 pixels×vertical 4pixels), motion compensation is performed on each of the blocks with adifferent motion vector, frequency transform is performed on adifference signal that is a difference with an original signal, thedifference signal is collected into a low frequency range andquantization is performed on the difference signals. Generally, atransform block size having a large size is advantageous becausecorrelation can be more effectively used.

CITATION LIST Non Patent Literature NPL 1

-   ISO/IEC 14496-10 “MPEG-4 Part 10 Advanced Video Coding”

SUMMARY OF INVENTION

In the conventional standards, however, a macroblock is partitioned intosmall blocks and motion compensation is performed on each of the smallblocks. Frequency transform is performed on each of the partitionedsmall blocks. More specifically, for example, when a macroblock ispartitioned into small blocks each composed of a unit of small block ofhorizontal 4 pixels×vertical 4 pixels, motion compensation and frequencytransform are performed by a unit of horizontal 4 pixels×vertical 4blocks. In this case, only correlation within a limited space can beused and it is difficult to collect the difference signals into a lowfrequency range by frequency transform. Therefore, there is a problemthat efficiency for reducing data amount by quantization is decreased,with a result that coding efficiency is decreased.

Meanwhile, when a block size for frequency transform is set to be largerthan a block size for motion compensation, a boundary of block formotion compensation is included in the block of frequency transform. Inthis case, since a steep edge is generated in a boundary of the motioncompensation block, it is difficult to collect the difference signalinto a low frequency range by frequency transform. Therefore, asdescribed above, there is a problem that efficiency for reducing data byquantization is decreased, with a result that coding efficiency isdecreased.

Therefore, the present invention is conceived to solve the abovementioned problem, and has an object to provide an image coding methodand an image decoding method for increasing coding efficiency when aprediction image of a target block is generated by performing predictionprocess on a prediction unit which is an area obtained by partitioningthe target block.

In order to solve the above described problem, an image coding methodaccording to an aspect of the present invention is an image codingmethod for coding an input image per block to generate a coded imagesignal, the image coding method including: performing prediction processon prediction units each of which is an area obtained by partitioning atarget block, to generate a prediction image of the target block;comparing, with one of the prediction units, a corresponding one oftransform units each of which is an area obtained by partitioning thetarget block and is a processing unit of frequency transform, to detectpart of a boundary of the prediction unit, the boundary being locatedwithin the transform unit; performing boundary filtering process on thedetected part of the boundary of the prediction unit in the generatedprediction image; calculating a difference between the prediction imageafter the filtering process and the input image, to generate adifference image of the target block; and performing frequency transformon the difference image for each of the transform units.

In this way, even if there is a part of a boundary of the predictionunit within the transform unit, filtering process can be performed onthe part of the boundary. With this, it is possible to smooth rapidvariations which occur in the part of the boundary of the predictionunit and to decrease a value of the difference image between the inputimage and the prediction image. Moreover, as similarly to theconventional standards, since it is not necessary to make the transformunit smaller such that there is no boundary within the transform unit,it is possible to extend the transform unit even if the prediction unitis small. As a result, since it is possible to significantly reduce acoding amount, it is possible to increase coding efficiency.

Moreover, in order to solve the above described problem, an imagedecoding method according to an aspect of the present invention is animage decoding method for decoding a coded image signal obtained bycoding an image per block, the image decoding method including:performing inverse transform for transform units each of which is anarea obtained by partitioning a target block, to generate a decodeddifference image of the target block; performing prediction process onprediction units each of which is an area obtained by partitioning thetarget block, to generate a prediction image of the target block;comparing one of the transform units with a corresponding one of theprediction units to detect part of a boundary of the prediction unit,the boundary being located within the transform unit; performing aboundary filtering process on the detected part of the boundary of theprediction unit in the generated prediction image; and adding theprediction image after the filtering process and the decoded differenceimage to generate a decoded image of the target block.

With this, it is possible to decode a coded image signal which is codedby the above described image coding method.

It should be noted that the present invention may be implemented as animage coding apparatus which includes processing units each of whichperforms a corresponding one of the steps included in the abovedescribed image coding method. Moreover, the present invention may beimplemented as an image decoding apparatus which includes processingunits each of which performs a corresponding one of the steps includedin the above described image decoding method.

With the present invention, it is possible to increase coding efficiencyby performing filtering process on a part of the boundary of theprediction unit located within the transform unit.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a block diagram showing a functional configuration of animage coding apparatus according to Embodiment 1 of the presentinvention.

FIG. 1B is a flowchart showing a flow of processing in the image codingmethod according to Embodiment 1 of the present invention.

FIG. 2 is a diagram showing an example of a method for partitioning thetarget block.

FIG. 3 is a schematic view for explaining motion compensation.

FIG. 4 is a flowchart showing a flow of a boundary filtering processaccording to Embodiment 1 of the present invention.

FIG. 5 is a diagram showing an example of a transform unit correspondingto a partition method.

FIG. 6 is a diagram for explaining the target boundary.

FIG. 7 is a schematic view for explaining characteristics of an imageincluding the target boundary.

FIG. 8 is a diagram for explaining filtering process.

FIG. 9A is a block diagram showing a functional configuration of animage decoding apparatus according to Embodiment 2 of the presentinvention.

FIG. 9B is a flowchart showing a flow of processing in the imagedecoding method according to Embodiment 2 of the present invention.

FIG. 10 is a flowchart showing a flow of boundary filtering processaccording to Embodiment 2 of the present invention.

FIG. 11 is a flowchart showing a flow of processing performed by amethod for determining a filtering process target pixel according toEmbodiment 3 of the present invention.

FIG. 12 is a schematic view for explaining an example of filteringprocess according to Embodiment 4 of the present invention.

FIG. 13 is a block diagram showing a functional configuration of animage coding apparatus according to Embodiment 5 of the presentinvention.

FIG. 14 is a flowchart showing an operation of a boundary filtering unitaccording to Embodiment 5 of the present invention.

FIG. 15 is a schematic view showing an example of a transform sizecandidate for a transform unit with respect to a partition method.

FIG. 16 is a flowchart showing a flow of processing performed by amethod for determining a transform size.

FIG. 17 is a block diagram showing a functional configuration of animage decoding apparatus according to Embodiment 6 of the presentinvention.

FIG. 18 is a flowchart showing a flow of filter information decodingprocesses according to Embodiment 6 of the present invention.

FIG. 19 is a flowchart showing a flow of decoding processing wheninformation of a transform unit is included in filter information.

FIG. 20 is a block diagram of a code sequence in the image coding methodaccording to Embodiment 7 of the present invention.

FIG. 21 is a schematic view for explaining an example of how to describefilter information on the code sequence.

FIG. 22 is a block diagram showing a functional configuration of animage coding apparatus according to Embodiment 8 of the presentinvention.

FIG. 23 is a block diagram showing a functional configuration of animage decoding apparatus according to Embodiment 9 of the presentinvention.

FIG. 24 is an overall configuration of a content providing system whichimplements content distribution services.

FIG. 25 is an overall configuration of a digital broadcasting system.

FIG. 26 is a block diagram showing an example of a configuration of atelevision.

FIG. 27 is a block diagram illustrating an example of a configuration ofan information reproducing/recording unit that reads and writesinformation from and on a recording medium that is an optical disk.

FIG. 28 is a diagram showing a configuration of a recording medium thatis an optical disk.

FIG. 29 is a diagram showing a structure of multiplex data.

FIG. 30 is a diagram showing how to multiplex each stream in multiplexdata.

FIG. 31 is a diagram showing how a video stream is stored in a stream ofPES packets in more detail.

FIG. 32 is a diagram showing a structure of TS packets and sourcepackets in the multiplexed data.

FIG. 33 is a diagram showing a data structure of a PMT.

FIG. 34 is a diagram showing an internal structure of multiplexed datainformation.

FIG. 35 is a diagram showing an internal structure of stream attributeinformation.

FIG. 36 is a diagram showing steps for identifying video data.

FIG. 37 is a block diagram showing an example of a configuration of anintegrated circuit for implementing the moving picture coding method andthe moving picture decoding method according to each of the embodiments.

FIG. 38 is a diagram showing a configuration for switching betweendriving frequencies.

FIG. 39 is a diagram showing steps for identifying video data andswitching between driving frequencies.

FIG. 40 is a diagram showing an example of a look-up table in whichvideo data standards are associated with driving frequencies.

FIG. 41, in (a), is a diagram showing an example of a configuration forsharing a module of a signal processing unit, while FIG. 41, in (b), isa diagram showing another example of a configuration for sharing amodule of the signal processing unit.

DETAILED DESCRIPTION OF INVENTION Embodiment 1

FIG. 1A is a block diagram showing a functional configuration of animage coding apparatus 100 according to Embodiment 1 of the presentinvention. The image coding apparatus 100 codes an input image for eachof the blocks to generate a coded image signal.

As shown in FIG. 1A, the image coding apparatus 100 includes a framememory 101, a motion estimation unit 102, a reference picture memory103, a coding control unit 104, a boundary filtering unit 105, adifference unit 106, a transformation unit 107, a quantization unit 108,a variable length coding unit 109, an inverse quantization unit 110, aninverse transformation unit 111, and an addition unit 112.

The frame memory 101 holds an input image. Moreover, the referencepicture memory 103 holds a reconstructed image. It should be noted thatthe frame memory 101 and the reference picture memory 103 may be anexternal memory connected to the image coding apparatus 100. In otherwords, the image coding apparatus 100 does not have to include the framememory 101 and the reference picture memory 103. Moreover, the imagecoding apparatus 100 does not have to include, as a separate memory,each of the frame memory 101 and the reference picture memory 103. Forexample, the frame memory 101 and the reference picture memory 103 maybe configured by sharing a memory.

The motion estimation unit 102 performs motion estimation for each ofthe prediction units to obtain a motion vector MV for each of theprediction units. Here, the prediction unit is an area which is obtainedby partitioning the target block, and is a processing unit of theprediction. The obtained motion vector MV is output to the codingcontrol unit 104 and the variable length coding unit 109.

The coding control unit 104 performs motion compensation on each of theprediction units based on the obtained motion vector, to generate amotion compensated image of the target block as a prediction image. Morespecifically, the coding control unit 104 first, for each predictionunit, reads a reference image from the reference picture memory 103based on the motion vector of the prediction unit. Then, the codingcontrol unit 104 merges the read reference images each corresponding toone of the prediction units to generate the motion compensated image.

In this way, a prediction image is generated by the motion estimationunit 102 and the coding control unit 104. In other words, the motionestimation unit 102 and the coding control unit 104 perform predictionprocess on each of the prediction units to generate the prediction imageof the target block.

Moreover, the coding control unit 104 outputs the generated predictionimage (motion compensated image) MC to the boundary filtering unit 105.Furthermore, the coding control unit 104 generates a partition method ICfor obtaining a prediction unit by partitioning the target block, andthen outputs IC to the variable length coding unit 109.

The boundary filtering unit 105 compares the transform unit with theprediction unit to detect a part of a boundary of the prediction unitwhich is located within the transform unit. The boundary filtering unit105 performs filtering process on the part of the detected boundary ofthe prediction unit in the generated prediction image.

Here, the transform unit is an area which is obtained by partitioningthe target block, and is a processing unit of the frequency transform.Moreover, the filtering process on the boundary is a processing forcorrecting a pixel value for a pixel adjacent to the boundary to smoothvariations in pixel value in a direction perpendicular to the boundary.

It should be noted that it is favorable that the boundary filtering unit105 does not perform filtering process on the boundary of the transformunit. In other words, it is favorable that the boundary filtering unit105 does not perform filtering process on the boundary overlapping withthe boundary of the transform unit, even if it is the boundary of theprediction unit. This is because generally, deblocking filtering isperformed on the boundary of the transform unit when the referencepicture is generated.

A filtered prediction image PR is output to the difference unit 106 andthe addition unit 112.

The difference unit 106 calculates a difference between the filteredprediction image and the input image to generate a difference image ofthe target block. More specifically, the difference unit 106 calculates,for each of the pixels of the target block, a difference value in pixelvalue between the filtered prediction image and the input image, andgenerates a difference image which holds the calculated difference valueas the pixel value. Then, the difference image RS is output to thetransformation unit 107.

The transformation unit 107 performs frequency transform on thedifference image for each of the transform units. The frequencytransform is, for example, orthogonal transform represented by DiscreteCosine Transform (DCT). Transform coefficients generated in thefrequency transform are output to the quantization unit 108.

The quantization unit 108 performs quantization on the transformcoefficients generated by the frequency transform to generate quantizedcoefficients QC. The generated quantized coefficients QC are output tothe variable length coding unit 109 and the inverse quantization unit110.

The variable length coding unit 109 performs variable length coding onthe quantized coefficients QC, the motion vector MV, and the partitionmethod IC to generate a coded image signal.

The inverse quantization unit 110 performs inverse quantization on thequantized coefficients generated by the quantization unit 108. Moreover,the inverse transformation unit 111 performs inverse frequency transformon the result of inverse quantization by the inverse quantization unit110, to generate a decoded difference image. Then, the decodeddifference image DR generated in this way is output to the addition unit112.

The addition unit 112 adds the decoded difference image and theprediction image to generate a reconstructed image. Then, thereconstructed image DC generated in this way is output to the referencepicture memory 103. In other words, the reconstructed image DC is heldin the reference picture memory 103 to be used as the reference imagewhen the subsequent pictures are coded.

Next, an image coding method performed by the above configured imagecoding apparatus 100 will be described.

FIG. 1B is a flowchart showing a flow of processing in the image codingmethod according to Embodiment 1 of the present invention.

Before the flowchart shown in FIG. 1B is started, the input image isheld in the frame memory 101. Moreover, a decoded image of the codedimage (reconstructed image) is already stored in the reference picturememory 103, and the reconstructed image is used as a reference picturewhen the target block is coded.

The target block which is held in frame memory 101 is partitioned intoprediction units according to a partition method instructed by thecoding control unit 104. Hereinafter, the prediction unit is referred toas a partitioned area.

FIG. 2 is a diagram showing an example of a method for partitioning thetarget block according to Embodiment 1 of the present invention.

In the present embodiment, as shown in (a) to (d) of FIG. 2, the targetblock is partitioned into a partitioned area of 8×8 pixels, 4×4 pixels,8×2 pixels, or 2×8 pixels. In other words, the partition methodinstructed by the coding control unit 104 is composed of four types,that is, (a) to (d) of FIG. 2.

In this state, prediction process is performed on each of thepartitioned areas and a prediction image of the target block isgenerated (Step S101). Step S101 corresponds to a prediction step.

More specifically, the motion estimation unit 102 performs motionestimation for each of the partitioned areas to obtain a motion vectorfor each of the partitioned areas (Step S101 a). More specifically, themotion estimation unit 102 performs, for each of the images of thepartitioned areas, motion estimation with respect to the referencepicture held in the reference picture memory 103. In the motionestimation, in a predetermined range in the reference picture (forexample, within a range of a rectangle having plus or minus 32 pixels ina horizontal direction and plus or minus 24 pixels in a verticaldirection), a position in the reference picture in which an evaluationvalue with respect to the image of the partitioned area, for example, isminimum is estimated. The motion estimation unit 102 obtains a motionvector from the position obtained in this way.

Here, the evaluation value is a value for determining the referenceimage. As the evaluation value, for example, an absolute value sum ofdifferences of pixel values between pixels corresponding to each othercan be used. Moreover, as the evaluation value, for example, a weightedsum of the absolute value sum of differences and a code amount of themotion vector and the like can be used.

The motion estimation unit 102, for each of the partition methods,performs motion estimation with respect to an image for each of thepartitioned areas, to obtain a motion vector for each of the partitionedareas. Furthermore, the motion estimation unit 102 determines theoptimum partition method (for example, a method in which a sum of theevaluation values with respect to each of the partitioned areas isminimum when the minimum is used as a basis for an optimum value). Then,the motion estimation unit 102 outputs the determined partition methodto the coding control unit 104 and outputs the motion vector MV for eachof the partitioned areas which is partitioned by a corresponding one ofthe partition methods to the variable length coding unit 109. Moreover,the determined partition method and the determined motion vector arenotified to the coding control unit 104.

Next, the coding control unit 104 generates, as a prediction image, themotion compensated image of the target block based on the motion vectorobtained by the motion estimation unit 102 (Step S101 b). Morespecifically, the coding control unit 104 reads the reference picturewith respect to each of the partitioned areas from the reference picturememory 103 and merges the read reference images, to generate theprediction image. The generated prediction image MC is output to theboundary filtering unit 105.

For example, when, in FIG. 3, a target block 300 is partitioned into apartitioned area 301 and a partitioned area 302, the coding control unit104 obtains a reference image 303 and a reference image 304 from thereference picture, to generate a prediction image of the target block300 (motion compensated image). The prediction image is processed on aprocessing block-by-processing block basis (for example, a unit ofhorizontal 16 pixels and vertical 16 pixels).

Next, the boundary filtering unit 105 performs filtering process on theboundary of the partitioned area in the generated prediction image (StepS102). The operation of the boundary filtering unit 105 will bedescribed with reference to FIGS. 4 to 7 in detail.

FIG. 4 is a flowchart showing a flow of a boundary filtering processaccording to Embodiment 1 of the present invention. More specifically,FIG. 4 is a flowchart for describing the operation in which the boundaryfiltering unit 105 performs filtering process on the prediction imagebased on the partition method obtained from the coding control unit 104.

The boundary filtering unit 105 obtains, from the coding control unit104, a prediction image MC and the partition method corresponding to theprediction image MC (Step S401).

Next, the boundary filtering unit 105 determines whether or not theboundary of the partitioned area exists within the transform unit (StepS402). In other words, the boundary filtering unit 105 compares thetransform unit with the partitioned area, to detect the boundary of thepartitioned area which is located within the transform unit (alsoreferred to as target boundary).

More specifically, the boundary filtering unit 105 obtains informationabout the shape of the partitioned area based on the partition methodcorresponding to the prediction image MC. Then, the boundary filteringunit 105, for example, as shown in FIG. 5, compares the transform unitwhich is determined in advance according to the partition method withthe partitioned area, and determines whether or not the boundary of thepartitioned area exists within the transform unit.

With reference to FIG. 6, the comparison process in Step S402 will bedescribed in more detail. FIG. 6 shows, in (a), the boundary of thetransform unit in the target block. FIG. 6 shows, in (b), the boundaryof the partitioned area in the target block. Moreover, FIG. 6 shows, in(c), the target boundary which is detected by comparison between theboundary of the transform unit and the boundary of the partitioned area.In (c) of FIG. 6, the target boundary is illustrated by hatching.

For example, as shown in FIG. 5, when the partition method correspondsto the transform unit on a one-to-one basis, the boundary filtering unit105 compares, with the transform unit, the minimum partition width withrespect to the partition method. When the minimum partition width issmaller than a length of a side of the transform unit, the boundaryfiltering unit 105 determines that the boundary of the partitioned areaof the reference image MC is generated within the transform unit.

Here, when the target boundary is not detected (No in Step S402),filtering process is not performed on the prediction image of the targetblock and the boundary filtering process is ended.

Meanwhile, the target boundary is detected (YES in Step S402), theboundary filtering unit 105 performs filtering process on the targetboundary (for example, the target boundary shown in (c) of FIG. 6) (StepS403). Here, with reference to FIGS. 7 and 8, the filtering process onthe target boundary in Step S403 will be described in detail.

FIG. 7 is a schematic view for explaining characteristics of an imageincluding the target boundary.

FIG. 7 shows, in (a), an original image 701, a prediction image 702, anda difference image 706. The original image 701 is an image of the targetblock which is included in the input image held in the frame memory 101.The prediction image 702 is an image which is obtained by performingmotion compensation on each of the partitioned areas 703, 704, and 705.The difference image 706 is an image showing the difference between theoriginal image 701 and the prediction image 702.

As shown in the prediction image 702, when motion compensation isperformed on each of the partitioned areas, there is often difference inimage characteristics between the partitioned areas 703, 704, and 705.Therefore, in the difference image 706 that is a difference signalbetween the original image 701 and the prediction image 702, a pixelvalue of a pixel adjacent to the boundary of the partitioned area tendsto be great.

In (b) of FIG. 7, variations in pixel values in a portion 707 whichincludes an object boundary and crosses a partitioned area in thedifference image 706 are shown. Moreover, in (c) of FIG. 7, variationsin pixel values in a portion 708 which includes an object boundary likethat in the portion 707 and does not cross a partitioned area are shown.In (b) and (c) of FIG. 7, a horizontal axis shows positions of pixels ina direction perpendicular to the boundary of the partitioned area, whilea vertical axis shows pixel values.

In (b) of FIG. 7, there are great variations in value caused by theboundary of the partitioned area in the central part and there are smallvariations in value caused by the object boundary near the right side.In (c) of FIG. 7, there are only small variations in value caused by theobject boundary near the right side.

Here, when two areas 709 circled by thick lines shown in (a) of FIG. 7are each a transform unit, transform is performed on an image having alarge difference value as shown in (b) of FIG. 7. In orthogonaltransform represented by Discrete Cosine Transform (DCT) for use inimage coding, it is possible to efficiently transform an image which isgreatly included in a natural image and includes spatial variations insmall spatial pixel value (an image which includes spatial variations inpixel value caused by the object boundary of the natural image shown in(c) of FIG. 7). However, it is impossible to efficiently transform animage which includes an edge indicating steep variations in pixel valueas shown in (b) of FIG. 7.

Therefore, in Step S402, the boundary filtering unit 105 detects atarget boundary 710 shown in (d) of FIG. 7. The boundary filtering unit105 performs filtering process on an image adjacent to the targetboundary 710 in the prediction image. In other words, the boundaryfiltering unit 105 performs filtering process on the target boundary.

Here, the image which is subject to the filtering process is not adifference image but a prediction image. This is because when thefiltering process is performed on the difference image between the inputimage and the prediction image, it is difficult to restore a componentwhich is lost in the filtering process. For example, when the filteringprocess is performed such that a frequency component of a frequencyhigher than a certain level in the difference image is replaced to zero,it is difficult to restore the frequency component of the high frequencywhich is lost in the filtering process even if quantization is performedwith significantly high precision on the filtered difference image. Inother words, it is difficult to restore an image similar to the inputimage.

Meanwhile, when the filtering process is performed on the predictionimage, quantization is performed on transform coefficients obtained byfrequency transform on the difference image between the filteredprediction image and the input image. Therefore, when quantization isperformed with significantly high precision, it is possible to restorean image similar to the input image.

As described above, the filtering process has an object to smooth thesteep edge which is not included in the natural image. With reference toFIG. 8, filtering process to be performed on an image adjacent to thetarget boundary will be described.

FIG. 8 is a diagram for explaining the filtering process. In FIG. 8,pixel values of pixels adjacent to the target boundary are shown. Here,a horizontal axis shows positions of pixels in a direction perpendicularto the target boundary, while a vertical axis shows pixel values. Thepixel values of pixel p3, pixel p2, pixel p1, pixel p0, pixel q0, pixelq1, pixel q2, and pixel q3 are p3, p2, p1, p0, q0, q1, q2, and q3,respectively.

In (a) of FIG. 8, pixel values of the input image are shown. In (b) ofFIG. 8, pixel values of the prediction image are shown. In (c) of FIG.8, pixel values of the prediction image on which the filtering processis performed on pixels adjacent to the target boundary in (b) of FIG. 8are shown.

As shown in (a), (b), and (c) of FIG. 8, the filtered prediction imageis closer to the input image compared with the reference image beforethe filtering process. As a result, since a difference value included inthe difference image is small, it is possible to efficiently performtransform and perform efficient coding.

As described above, the boundary filtering unit 105 performs filteringprocess on the adjacent pixel of the target boundary in a predeterminedmethod for smoothing variations in pixel value. The boundary filteringunit 105 outputs the filtered prediction image PR to the difference unit106 and the addition unit 112.

The description of FIG. 1B is resumed.

The difference unit 106 calculates a difference value between the pixelvalues of pixels corresponding each to the input image and theprediction image PR, to generate a residual image RS (Step S103). Theresidual image RS corresponds to the difference image.

The transformation unit 107 performs frequency transform on the residualimage RS (Step S104). The transformation unit 107 outputs the transformcoefficients obtained by the frequency transform to the quantizationunit 108.

A method for determining the adjacent pixel of the target boundary whichis subject to the filtering process will be described in detail inEmbodiment 3, while a method for performing filtering process on theadjacent pixel will be described in detail in Embodiment 4.

As described above, the image coding apparatus 100 according toEmbodiment 1 of the present invention, when the input image is coded,partitions the input image into blocks (units of coding) each of whichis a square area and a processing unit for coding. The image codingapparatus 100 partitions the partitioned blocks further into a pluralityof partitioned areas in a predetermined partition method. Then, theimage coding apparatus 100 performs motion estimation and motioncompensation on each of the partitioned areas to generate a predictionimage of the target block. Here, the image coding apparatus 100 detects,as the target boundary, a part of the boundary of the partitioned areawhich does not match the transform unit which is a processing unit fortransform process on the difference signal in the subsequent stage.Then, the image coding apparatus 100 performs filtering process on thetarget boundary in the prediction image.

With this operation, even when a part of a boundary of a prediction unit(partitioned area) exists within the transform unit, the filteringprocess can be performed on the part of the boundary. Therefore, it ispossible to smooth rapid variations in pixel value which occur in thepart of the boundary of the prediction unit and to decrease a value ofthe difference image between the input image and the prediction image.Moreover, as similarly to the conventional standards, since it is notnecessary to make the transform unit smaller such that there is noboundary of the prediction unit (partitioned area) within the transformunit, it is possible to extend the transform unit even when theprediction unit is small. As a result, since it is possible tosignificantly reduce a code amount, it is possible to increase codingefficiency.

In the present embodiment, the filtering process is performed on only apart of the boundary of the prediction unit which is located within thetransform unit. Therefore, processing amount can be reduced comparedwith the case where the filtering process is performed on the whole ofthe boundary of the prediction unit. Furthermore, in the decoded imageor the reference image, it is possible to reduce the filtering processwhich overlaps with the process performed on the boundary of thetransform unit by a coding distortion removal filter (deblockingfilter), and it is possible to reduce degradation in image quality(image blur) caused by a plurality of times of filtering process.

It should be noted that the partition method and the transform unit donot have to be limited to the partition method and the transform unitshown in FIG. 5. For example, even when a part of the prediction unitoverlaps with a part of the transform unit, the image coding methodaccording to the present embodiment obtains the effect similar to theabove described effect by performing filtering process on the boundaryof the prediction unit within the transform unit.

It should be noted that the image coding apparatus 100 according to thepresent embodiment performs filtering process on the whole of theboundary of the partitioned area within the transform unit. However, itis not necessary to perform the filtering process on the whole of thepartitioned area within the transform unit. For example, when areference image is obtained from an area in which two partitioned areashave the same motion vector and is adjacent to the partitioned areas ina reference picture for each of two partitioned areas, filtering processmay be skipped for the boundary between the two partitioned areas. Withthis, the amount of filtering process can be reduced and an excessivefiltering process on the prediction image can be restricted, anddegradation in image quality caused by the filtering process (imageblur) can be restricted.

It should be noted that the information indicating a skip of thefiltering process may be described as stream header information. Withthis, since ON/OFF of the filter can be controlled according to imagecharacteristics, it is possible to maintain image quality of the decodedimage at a high level. It should be noted that how to transmit filterheader information will be described in detail in other embodiments.

It should be noted that in the prediction of FIG. 1B (Step S101), intraprediction process may be performed instead of the motion estimation(Step S101 a) and the motion compensation (Step S101 b). In other words,in the prediction process (Step S101), the coding control unit 104 mayperform intra prediction process for each of the prediction units basedon the reconstructed image of the already coded block, to generate theintra-prediction image of the target block as the prediction image. Inthis case, the image coding apparatus 100 does not have to include themotion estimation unit 102.

It should be noted that the image coding apparatus 100 does notnecessarily have to include all the processing units shown in FIG. 1A.For example, the image coding apparatus 100 may include only theprocessing units each of which performs a corresponding one of the stepsincluded in the image coding method shown in FIG. 1B.

Embodiment 2

FIG. 9A is a block diagram showing a functional configuration of animage decoding apparatus 900 according to Embodiment 2 of the presentinvention. The image decoding apparatus 900 decodes a coded image signalin which the target image is coded, to generate a decoded image. Theimage decoding apparatus 900 includes a variable length decoding unit901, an inverse quantization unit 902, an inverse transformation unit903, an addition unit 904, a decoding control unit 905, a boundaryfiltering unit 906, a motion compensation unit 907, and a referencepicture memory 908.

In the present embodiment, an input code sequence BS is determined to bea coded image signal generated by the image coding apparatus 100according to Embodiment 1 of the present invention. In other words, theinput code sequence BS is determined to be a coded image signal in whichthe input image is coded for each of the blocks.

The reference picture memory 908 holds a reconstructed image that is analready decoded image. It should be noted that the reference picturememory 908 may be an external memory connected to the image decodingapparatus 900. In other words, the image decoding apparatus 900 does notnecessarily have to include the reference picture memory 908.

The variable length decoding unit 901 performs variable length decodingon the input code sequence BS to obtain quantized coefficients, apartition method, and a motion vector. The partition method is a methodfor partitioning the target block into a plurality of prediction units(partitioned areas). The obtained quantized coefficients QC are outputto the inverse quantization unit 902, the partition method IC is outputto the decoding control unit 905, and the motion vector MV is output tothe motion compensation unit 907.

The inverse quantization unit 902 performs inverse quantization on thequantized coefficients. Moreover, the inverse transformation unit 903performs, for each of the transform units, inverse frequency transformon the result of inverse quantization in the inverse quantization unit902, to generate a decoded difference image. Then, the decodeddifference image DR generated in this way is output to the addition unit904.

The decoding control unit 905 obtains the partition method IC for thetarget block, and outputs IC to the boundary filtering unit 906 and themotion compensation unit 907.

The motion compensation unit 907 obtains a partition method forpartitioning the target block into prediction units. Furthermore, themotion compensation unit 907 obtains a motion vector for each of theprediction units which are obtained by partitioning according to theobtained partition method. The motion compensation unit 907 performsmotion compensation on each of the prediction units based on theobtained motion vector, to generate a motion compensated image of thetarget block as a prediction image. In other words, the motioncompensation unit 907 performs prediction process on each of theprediction units which are areas obtained by partitioning the targetblock, to generate a prediction image of the target block.

In other words, the motion compensation unit 907 obtains, based on thepartition method and the motion vector, the reference image from thereference picture memory 908, to generate a motion compensated image forthe target block. Here, when the target block is partitioned, a motionvector is described for each of the partitioned areas (predictionunits). Therefore, the motion compensation unit 907, according to themotion vector, obtains a reference image for each of the partitionedareas and generates a motion compensated image by merging the obtainedreference images.

For example, as shown in FIG. 3, the target block 300 is partitionedinto the partitioned area 301 and the partitioned area 302, the motioncompensation unit 907 obtains the reference image 303 and the referenceimage 304 from the reference picture memory 908, and generates themotion compensated image as a prediction image MC. Then, the motioncompensation unit 907 outputs the prediction image MC to the boundaryfiltering unit 906.

The boundary filtering unit 906 compares the transform unit with theprediction unit to detect a part of a boundary of the prediction unitwhich is located within the transform unit. The boundary filtering unit906 performs filtering process on the detected part of the boundary ofthe prediction unit in the generated prediction image.

It should be noted that the transform unit, for example, may bedetermined according to a predetermined correspondence relationshipbetween the prediction unit and the transform unit. Moreover, forexample, the transform unit may be determined independently of theprediction unit.

The addition unit 904 adds the filtered prediction image and the decodeddifference image to generate a decoded image of the target block.

Next, an image coding method performed by the above configured imagedecoding apparatus 900 will be described.

FIG. 9B is a flowchart showing a flow of processing in the imagedecoding method according to Embodiment 2 of the present invention.

Before the flowchart shown in FIG. 9B is started, a decoded image isalready stored in the reference picture memory 908. The decoded image isused as a reference picture when a code sequence is decoded.

First, the inverse transformation unit 903 performs inverse frequencytransform on each of the transform units to generate a decodeddifference image of the target block (Step S901).

Next, the motion compensation unit 907 performs prediction process oneach of the prediction units that are areas obtained by partitioning thetarget block, to generate the prediction image of the target block (StepS902). More specifically, the motion compensation unit 907 first obtainsa motion vector for each of the prediction units (Step S902 a). Next,the motion compensation unit 907 performs motion compensation on each ofthe prediction units based on the obtained motion vector, to generate amotion compensated image of the target block as the prediction image(Step S902 b).

The boundary filtering unit 906 performs filtering process on theboundary of the prediction unit in the generated prediction image (StepS903). The operation of the boundary filtering unit 906 will bedescribed with reference to FIG. 10.

FIG. 10 is a flowchart showing a flow of boundary filtering processaccording to Embodiment 2 of the present invention. More specifically,FIG. 10 is a flowchart for explaining the operation in which theboundary filtering unit 906 performs filtering process on the predictionimage based on the partition method obtained from the decoding controlunit 905.

The boundary filtering unit 906 obtains the prediction image of thetarget block from the motion compensation unit 907, and further obtainsthe partition method of the target block from the decoding control unit905 (Step S1001).

Next, the boundary filtering unit 906 determines whether or not part ofthe boundary of the partitioned area exists within the transform unit(Step S1002). In other words, the boundary filtering unit 105 comparesthe transform unit with the partitioned area to detect the part of theboundary of the partitioned area which is located within the transformunit.

More specifically, the boundary filtering unit 906 obtains informationabout the shape of the partitioned area based on the partition method.The boundary filtering unit 906, for example, as shown in FIG. 5,compares, with the partitioned area, the transform unit which isdetermined in advance according to the partition method, and determineswhether or not the boundary of the partitioned area exists within thetransform unit. The comparison process is similar to that described inStep S402 in Embodiment 1, and a detailed description will be omitted.

Here, when the target boundary is not detected (No in Step S1002),filtering process is not performed on the prediction image of the targetblock and the boundary filtering process is ended.

Meanwhile, when the target boundary is detected (YES in Step S1002), theboundary filtering unit 906 performs filtering process on the targetboundary (for example, the target boundary shown in (c) of FIG. 6) (StepS1003). The filtering process in Step S1003 is similar to the filteringprocess described in Step S403 in Embodiment 1, and a detaileddescription will be omitted. The boundary filtering unit 105 outputs thefiltered prediction image PR to the addition unit 904.

Next, the addition unit 904 adds the decoded difference image DR and thefiltered prediction image PR, to generate a decoded image DC of thetarget block (Step S904). The decoded image DC generated in this way isoutput to the reference picture memory 908. In other words, the decodedimage DC is an output image and a reference picture which is used in thesubsequent coding.

As described above, the image decoding apparatus 900 according toEmbodiment 2 of the present invention obtains partition information foreach of the target blocks when decoding the code sequence generated bythe image coding method according to Embodiment 1 of the presentinvention. The image decoding apparatus 900 performs motion compensationon each of the prediction units according to the partition method, andone of a motion vector which already described in the code sequence or asurrounding motion vector is already decoded, to generate the motioncompensated image of the target block as the prediction image. In theprediction image generated in this way, filtering process is performedon the boundary of the partitioned area which is located in theprediction unit. The image decoding apparatus 900 adds the predictionimage after the filtering and the decoded difference image to generate adecoded image.

With this operation, the code sequence generated by the image codingmethod according to Embodiment 1 of the present invention can beaccurately decoded.

It should be noted that the partition method and the transform unit donot have to be limited to the partition method and the transform unitshown in FIG. 5. When the boundary of the portioned area is locatedwithin the transform unit, the effect similar to the above describedeffect can be obtained.

It should be noted that the image decoding apparatus 900 according tothe present embodiment, as similarly to the image decoding apparatusaccording to Embodiment 1, performs filtering process on the whole ofthe boundary of the partitioned area within the transform unit. However,the image decoding apparatus 900 does not necessarily have to performthe filtering process on the whole of the partitioned area within thetransform unit. For example, when a reference image is obtained from anarea in which two partitioned areas have the same motion vector and isadjacent to the partitioned areas in a reference picture for each of twopartitioned areas, filtering process may be skipped for the boundarybetween the two partitioned areas. With this, the amount of filteringprocess can be reduced and an excessive filtering process on theprediction image can be restricted, and degradation in image qualitycaused by the filtering process (image blur) can be restricted.

It should be noted that when the information indicating a skip of thefiltering process is described in stream header information, ON/OFF ofthe filter can be controlled according to image characteristics bycoding the information. Therefore, it is possible to maintain imagequality of the decoded image at a high level. It should be noted thathow to decode filter header information will be described in detail inother embodiments.

It should be noted that a method for determining the adjacent pixel ofthe target boundary which is subject to filtering process, as similarlyto Embodiment 1, will be described in detail in Embodiment 3, while amethod for performing filtering process on the adjacent pixel will bedescribed in detail in Embodiment 4.

It should be noted that the image decoding apparatus 900 does notnecessarily have to include all the processing units shown in FIG. 9A.For example, the image decoding apparatus 900 may include only theprocessing units each of which performs a corresponding one of the stepsincluded in the image decoding method shown in FIG. 9B.

Embodiment 3

In Embodiment 3 of the present invention, filtering process on thetarget boundary in Embodiments 1 and 2 will be described. Especially, amethod for determining a filtering process target pixel which is a pixelthat is subject to filtering process will be described in detail.

Here, a portion on which filtering process is performed is a boundary ofthe prediction unit which is located within the transform unit (targetboundary). At this time, a method for determining the filtering processtarget pixel that is a pixel adjacent to the target boundary and issubject to the filtering process will be described with reference topixels p3, p2, p1, p0, q0, q1, q2, and q3 arranged in a row in thefiltering process direction and shown in FIG. 8. It should be noted thepixel values of pixel p3, pixel p2, pixel p1, pixel p0, pixel q0, pixelq1, pixel q2, and pixel q3 are p3, p2, p1, p0, q0, q1, q2, and q3,respectively.

FIG. 11 is a flowchart showing a flow of processing performed under amethod for determining a filtering process target pixel according toEmbodiment 3 of the present invention. First, the boundary filteringunit 105 or 906 (hereinafter, the boundary filtering unit 105 will bedescribed on behalf of the boundary filtering unit 906) obtains filterinformation indicating ON/OFF of the filter which is described in thestream header information. The boundary filtering unit 105 determineswhether or not the filter information is information indicating filterOFF (Step S1101).

Here, when the filter information is information indicating filter OFF(YES in Step S1101), the boundary filtering unit 105 determines thatnone of pixels adjacent to the target boundary are filtering processtarget pixels (Step S1102). Meanwhile, when the filter information isinformation indicating filter ON (NO in Step S1101), the boundaryfiltering unit 105 determines that the pixel p0 and the pixel q0 arefiltering process target pixels (Step S1103). Next, the boundaryfiltering unit 105 compares an absolute value of the difference in pixelvalue between the pixel p1 and the pixel p0 with a predeterminedthreshold TH (Step S1104). Here, the threshold TH is a value given byExpression 1, for example.[Math. 1]TH=0.8·(2^((QP+Offset)/6)+1)  (Expression 1)

Here, QP is a quantization parameter which denotes quantizationprecision. Moreover, Offset is described in the file header informationas an adjustment parameter.

It should be noted that it may be determined such that when thequantization parameter QP is greater, the threshold TH is greater. It isnot necessary to be determined according to Expression 1. Moreover, thethreshold TH may be described in the file header information.

Here, the absolute value of the difference is smaller than the thresholdTH (YES in Step S1104), the boundary filtering unit 105 furtherdetermines that the pixel p1 is a filtering process target pixel (StepS1105). Meanwhile, the absolute value of the difference is greater thanor equal to the threshold TH (NO in Step S1104), the processing moves onto the next step S1106.

Next, the boundary filtering unit 105, as similarly to Step S1104,compares an absolute value of the difference in pixel value between thepixel p1 and the pixel q0 with a predetermined threshold TH (StepS1106). It should be noted that the threshold TH is used with the samevalue in Step S1104.

Here, the absolute value of the difference is smaller than the thresholdTH (YES in Step S1106), the boundary filtering unit 105 furtherdetermines that the pixel q1 is a filtering process target pixel (StepS1107) and then the determination process for the filtering processtarget pixel is ended. Meanwhile, the absolute value of the differenceis greater than or equal to the threshold TH (NO in Step S1106), thedetermination process for the filtering process target pixel is ended.

It should be noted that the threshold TH is a value which variesdepending on the quantization parameter QP indicated in Expression 1 asan example. However, the threshold TH may be a value which variesdepending on a difference in motion vector between prediction unitsadjacent to the target boundary. For example, it may be determined suchthat when a sum MVD of (i) the absolute value of the difference of the xcomponent of the motion vector and (ii) the absolute value of thedifference of the y-component of the motion vector is greater, thethreshold TH is greater.

When the motion vector difference is greater, variations in imagecharacteristics in the target boundary are probably different.Therefore, by varying the threshold TH depending on the motion vectordifference, the filtering process target pixel adapted to the imagecharacteristics can be determined.

Moreover, the threshold TH may be determined such that it variesdepending on the MVD and the quantization parameter QP.

With the above described method, the filtering process target pixel isdetermined.

It should be noted that the boundary filtering unit 105 determineswhether or not only the pixels p0, p1, q0, and q1 are filtering processtarget pixels. Furthermore, the boundary filtering unit 105 may make adetermination on the pixel p2 and the pixel q2. In that case, theboundary filtering unit 105 may determine that the pixel p2 and thepixel q2 are filtering process target pixels, by comparing the absolutevalue of the difference (for example, |p2−p1|) with the threshold on theassumption that the adjacent pixel is the filtering process target pixel(for example, p1).

It should be noted that in Step S1101, the case where the filterinformation indicating ON/Off of the filter is described in the streamheader information. However, the filter information does not necessarilyhave to be described in the header information. For example, if ON/OFFof the filter is determined in the coding method and the decodingmethod, the filter information does not have to be described in theheader information. In this case, the amount of the header informationcan be reduced.

Moreover, for example, the filter information indicating the number ofboundary pixels may be described in the header information. In thiscase, in Step S1101 of FIG. 11, the boundary filtering unit 105 maydetermine the filtering process target pixel according to the number ofpixels indicated by the filter information. For example, when the numberof pixels is one, the boundary filtering unit 105 may determine that thepixels p0 and q0 are filtering process target pixels. Moreover, forexample, when the number of pixels is two, the boundary filtering unit105 may determine that the pixels p1, p0, q0, and q1 are filteringprocess target pixels. With this, the amount of processing by theboundary filtering unit 105 can be significantly reduced.

It should be noted that how to transmit the header information will bedescribed in detail in other embodiments.

Embodiment 4

In Embodiment 4 of the present invention, filtering process on thetarget boundary in Embodiments 1 and 2 will be described. Morespecifically, the filtering process on the filtering process targetpixels determined in Embodiment 3 will be described in detail.

Here, it is assumed that the filtering process target pixels are alreadydetermined by the method shown in Embodiment 3. Moreover, the filteringprocess will be described with reference to pixel indexes p3, p2, p1,p0, q0, q1, q2, and q3 arranged in a line in the filtering processdirection shown in FIG. 8. It should be noted the pixel values of thepixel p3, the pixel p2, the pixel p1, the pixel p0, the pixel q0, thepixel q1, the pixel q2, and the pixel q3 are p3, p2, p1, p0, q0, q1, q2,and q3, respectively. Moreover, the following will describe the casewhere the image coding apparatus 100 according to Embodiment 1 performsthe filtering process.

FIG. 12 is a schematic view for explaining an example of filteringprocess according to Embodiment 4 of the present invention. In FIG. 12,a target block 1200 is partitioned into a partitioned area 1201 and apartitioned area 1202 each of which is a unit of horizontal 4 pixels andvertical 8 pixels. The target block 1200 is a transform unit, and thepartitioned area 1201 and the partitioned area 1202 are each aprediction unit.

In this case, the coding control unit 104 obtains a reference image 1203and a reference image 1204 from the reference picture memory 103. Thecoding control unit 104 merges the reference image 1203 and thereference image 1204 to generate a prediction image of the target block(motion compensated image).

For example, when the filtering process target pixels are the pixels p0and q0 in all lines, the boundary filtering unit 105 performs filteringprocess with the use of a reference image area 1205 adjacent to thereference image 1203 and a reference image area 1206 adjacent to thereference image 1204. For example, the boundary filtering unit 105determines that an average value obtained by the calculation ofoverlapped areas (cross-hatched parts in the target block 1200) by pixelunits is a filtered pixel value.

It should be noted that here, the boundary filtering unit 105 calculatesan average value of the two reference images to determine the pixelvalue of the pixel adjacent to the target boundary. The boundaryfiltering unit 105, however, may perform filtering process as shown inExpression 2.[Math. 2]p0=p0_(a) ·d ₀ +p0_(b)·(1−d ₀)q0=q0_(b) ·d ₀ +q0_(a)·(1−d ₀)p1=p1_(a) ·d ₁ +p1_(b)·(1−d ₁)q1=q1_(b) ·d ₁ +q1_(a)·(1−d ₁)p2=p2_(a) ·d ₂ +p2_(b)·(1−d ₂)q2=q2_(b) ·d ₂ +q2_(a)·(1−d ₂)  (Expression 2)

Here, d0, d1, and d2 are filter coefficients each of which is determinedby each of the distances from the target boundary, and they range from0.5 to 1. For example, d0 is equal to 0.5, d1 is equal to 0.75, and d2is equal to 0.875. These values may be determined in advance as thevalues common to the image coding method or the image decoding method,and may be described in the stream header information.

When filter strength is increased (smoothness is enhanced), the filtercoefficients d0, d1, and d2 may be each set to a value near 0.5.

It should be noted that the boundary filtering unit 105 may performfiltering process by up-shift or downshift according to Expression 3obtained by the transform of Expression 2. For example, d0=0.5, d1=0.75,and d2=0.875 in Expression 2 can be represented as A=1, B=1, C=1, D=3,E=2, F=1, G=7, H=3, and I=1 in Expression 3. In this way, when thefiltering process is performed according to Expression 3, the amount ofinformation of the header information can be reduced and the processingamount of the filtering process can be reduced.[Math. 3]p0=(p0_(a) ·A)>>C+(p0_(b) ·B)>>Cq0=(q0_(b) ·A)>>C+(q0_(a) ·B)>>Cp1=(p1_(a) ·D)>>E+(p1_(b) ·F)>>Eq1=(q1_(b) ·D)>>E+(q1_(a) ·F)>>Ep2=(p2_(a) ·G)>>H+(p2_(b) ·I)>>Hq2=(q2_(b) ·G)>>H+(q2_(a) ·I)>>H  (Expression 3)

By the method shown above, the boundary filtering unit 105 can performfiltering process on the determined filter target pixels. It should benoted that the boundary filtering unit 105 can reduce memory access byobtaining an image of an area larger than the partitioned area(prediction unit) when obtaining the reference image from the referencepicture memory 103.

It should be noted that the boundary filtering unit 105 may performfiltering process in a method other than the above described method. Theboundary filtering unit 105 may perform filtering process according toExpression 4, for example. Here, the filtered pixel values of the pixelq1, the pixel q0, the pixel p0, and the pixel p1 are expressed as q′1,q′0, p′0, and p′1, respectively.[Math. 4]p′0=c _(0,0) ·p1+c _(0,1) ·p0+c _(0,2) ·q0+c _(0,3) ·q1p′1=c _(1,0) ·p2+c _(1,1) ·p1+c _(1,2) ·p0+c _(1,3) ·q0q′0=c _(0,0) ·q1+c _(0,1) ·q0+c _(0,2) ·p0+c _(0,3) ·p1q′1=c _(1,0) ·q2+c _(1,1) ·q1+c _(1,2) ·q0+c _(1,3) ·q1  (Expression 4)

Here, c_(0,0) c_(0,1) c_(0,2) c_(0,3) c_(1,0) c_(1,1) c_(1,2) andc_(1,3) are filter coefficients, and for example, can be expressed byExpression 5.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack & \; \\{{\left\{ {c_{0,0},c_{0,1},c_{0,2},c_{0,3}} \right\} = \left\{ {\frac{1}{8},\frac{1}{2},\frac{1}{2},{- \frac{1}{8}},} \right\}}{\left\{ {c_{1,0},c_{1,1},c_{1,2},c_{1,3}} \right\} = \left\{ {\frac{7}{16},\frac{3}{16},\frac{5}{16},\frac{1}{16}} \right\}}} & \left( {{Expression}\mspace{14mu} 5} \right)\end{matrix}$

The filtering process implemented by the filter coefficients inExpression 5 smoothes variations in pixel value in the target boundary,as shown in a shift from the image in (b) of FIG. 8 to the image in (c)of FIG. 8. It should be noted that the filter coefficients are notlimited to Expression 5. For example, rounded values may be set as shownin Expression 6.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 6} \right\rbrack & \; \\{{\left\{ {c_{0,0},c_{0,1},c_{0,2},c_{0,3}} \right\} = \left\{ {\frac{1}{4},\frac{1}{4},\frac{1}{4},\frac{1}{4},} \right\}}{\left\{ {c_{1,0},c_{1,1},c_{1,2},c_{1,3}} \right\} = \left\{ {\frac{1}{4},\frac{1}{4},\frac{1}{4},\frac{1}{4}} \right\}}} & \left( {{Expression}\mspace{14mu} 6} \right)\end{matrix}$

In this case, the filtering process is strong (for enhancingsmoothness). When the filtering process is performed in this way, anobtaining amount of pixel data for motion compensation can be reducedand the processing amount can be reduced.

It should be noted that here, the case where a pixel whose distance fromthe target boundary is up to two pixels is the filter target pixel isdescribed. Filtering process similar to the above described process canbe performed even when a pixel whose distance from the target pixel isup to 3 pixels is the filtering process target pixel, by increasing thenumber of filter coefficients.

It should be noted that one of kinds of filters, filter coefficients, acode indicating filter strength may be described in the headerinformation. With this, a steep edge which does not exist in a naturalimage within the transform unit can be removed. As a result, since theprediction image can be closer to the input image, the difference imagecan be efficiently transformed by the transform, and the code amount canbe reduced.

It should be noted that the case is described where the target boundaryis a horizontal direction and a vertical direction. Process can beperformed similarly to the above described process also when the targetboundary is a diagonal direction.

It should be noted that how to transmit filter header information willbe described in detail in other embodiments.

Embodiment 5

Next, Embodiment 5 of the present invention will be described. In thepresent embodiment, the image coding apparatus determines the optimalfiltering process method among a plurality of filtering process methodsand describes the filter information indicating the determined filteringprocess method in a coded image signal.

FIG. 13 is a block diagram showing a functional configuration of animage coding apparatus according to Embodiment 5 of the presentinvention. As shown in FIG. 13, an image coding apparatus 1300 includesa frame memory 1301, a motion estimation unit 1302, a reference picturememory 1303, a coding control unit 1304, a boundary filtering unit 1305,a difference unit 1306, a transformation unit 1307, a quantization unit1308, a variable length coding unit 1309, an inverse quantization unit1310, an inverse transformation unit 1311, an addition unit 1312, and afilter information description unit 1313. The image coding apparatus1300 also includes the filter information description unit 1313 on topof the constituent elements included in the image coding apparatus 100in FIG. 1.

It should be noted that among the processing blocks in common with thoseof the image coding apparatus 100, a description will be omitted for theprocessing blocks excluding those of the boundary filtering unit 1305and the variable length coding unit 1309 because they operate similarlyto those in Embodiment 1.

The operation of the boundary filtering unit 1305 will be described withreference to FIG. 14. FIG. 14 is a flowchart for describing theoperation in which the boundary filtering unit 1305 performs filteringprocess on the prediction image MC based on the partition method.

The boundary filtering unit 1305 obtains, from the coding control unit1304, a prediction image MC and the partition method corresponding tothe prediction image MC (Step S1401).

Next, the boundary filtering unit 1305 determines whether or not theboundary of the partitioned area exists within the transform unit (StepS1402). In other words, the boundary filtering unit 1305 compares thetransform unit with the partitioned area to detect the boundary of thepartitioned area which is located within the transform unit.

More specifically, the boundary filtering unit 1305 obtains informationabout the shape of the partitioned area based on the partition methodcorresponding to the prediction image MC. The boundary filtering unit1305, for example, as shown in FIG. 5, compares, with the partitionedarea, the transform unit which is determined in advance according to thepartition method, and determines whether or not the boundary of thepartitioned area (target boundary) exists within the transform unit.Since the process of Step S1402 is similar to the process of Step S402in Embodiment 1, a detailed description will be omitted.

Here, when the target boundary is not detected (NO in Step S1402),filtering process is not performed on the prediction image of the targetblock and the boundary filtering process is ended.

Meanwhile, when the target boundary is detected (YES in Step S1402), theboundary filtering unit 1305 calculates an evaluation value for each ofthe filtering process methods for the target boundary (for example, thetarget boundary shown in (c) of FIG. 6) (Step S1403). Here, the boundaryfiltering unit 1305 finds an evaluation value for each of the filteringprocess methods described in Embodiment 4. There are four kinds offiltering process methods, for example, that is, no filter, thefiltering process by Expression 2 (coefficients (0.5, 0.7, and 0.825)),the filtering process by Expression 2 (coefficients (0.5, 0.5, and0.5)), and the filtering process by Expression 5. A combination of themethods is one example. For example, a combination of more methods isacceptable.

As an evaluation value, a weighted sum of (i) a sum of absolute valuesof the difference between the filtered prediction image and the inputimage and (ii) the code amount of the filtering process method and thefilter coefficients can be used. In this case, when the evaluation valueis smaller, the evaluation is higher.

Moreover, the evaluation value may be a sum of values obtained after thetransform process on the transform unit (for example, transform for usein coding or Hadamard transform). With this, the effect of filter can beappropriately evaluated.

The boundary filtering unit 1305, among a plurality of filtering processmethods, determines the filtering process method (method andcoefficient) having a value of the highest evaluation (here, thesmallest evaluation value). The boundary filtering unit 1305 outputs thefilter information indicating the determined filtering process method tothe filter information description unit 1313 (Step S1404).

When the determined filtering process method shows a result that nofiltering process is performed (NO in Step S1405), the boundaryfiltering unit 1305 directly outputs, without performing the filteringprocess, the motion compensated image (prediction image PR) to theaddition unit 1312 and the difference 1306. Meanwhile, when thefiltering process is performed (YES in Step S1405), the boundaryfiltering unit 1305 performs filtering process on the target boundarybased on the method determined by Step S1404 (Step S1406), and thenoutputs the filtered prediction image PR to the addition unit 1312 andthe difference unit 1306.

The filter information description unit 1313 outputs the filterinformation obtained from the boundary filtering unit 1305 as the filterheader information to the variable length coding unit 1309. It should benoted that the operation of the filter information description unit 1313and how to describe the filter header information will be described indetail in other embodiments.

By the image coding method according to the above described presentembodiment, it is possible to determine the filtering process methodhaving high filtering process effect among a plurality of the filteringprocess methods. According to the filtering process method that isdetermined in such a way, the filtering process can be performed on apart of the boundary of the prediction unit within the transform unit.As a result, since the difference between the input image and theprediction image can be smaller, the code amount can be reduced.

It should be noted that the boundary filtering unit 1305 calculates anevaluation value for each of all combinations of filtering processmethods. However, when the evaluation value is less than thepredetermined threshold, the boundary filtering unit 1305 may cancelcalculation of the subsequent evaluation values, and may determine thatthe filtering process method at that time is the optimal filteringprocess method. With this, the processing amount for determining thefiltering process method can be reduced.

Furthermore, as shown in FIG. 13, by a transfer of the transfer unit TUbetween the boundary filtering unit 1305 and the transformation unit1307, a change within the transform unit after considering the presenceor absence of the boundary filtering process can be implemented.

FIG. 15 is a diagram showing an example of a transform size candidatefor a transform unit with respect to a partition method. FIG. 16 is aflowchart showing a flow of processing performed by a method fordetermining the transform size.

First, the boundary filtering unit 1305 sequentially determines, withrespect to the transform sizes shown in FIG. 15, whether or not the partof the boundary of the partitioned area (prediction unit) indicated bythe partition method exists within the transform unit (Step S1601). Forexample, in the case of a transform size 8×8, three boundaries in ahorizontal direction are detected as the target boundaries and it isdetermined that the boundary of the partitioned area exists within thetransform unit.

Here, when the boundary of the partitioned area exists within thetransform unit (YES in Step S1601), the boundary filtering unit 1305determines, by the method shown in FIG. 14, the filtering process methodhaving an optimal evaluation value and holds the filtering processmethod and the evaluation value (Step S1602). Here, for example, whenthere are three boundaries of the partitioned area within the transformunit, the boundary filtering unit 1305 may determine the filteringprocess method based on a sum of evaluation values for the threeboundaries.

Meanwhile, when there is no boundary of the partition area within thetransform unit (NO in Step S1601), the boundary filtering unit 1305calculates an evaluation value and holds the evaluation value (StepS1603). For example, in the case of a transform size 2×2, since there isno boundary of the partitioned area within the transform unit, anevaluation value for the transform size 2×2 is calculated. Also in thiscase, a method similar to the above described method (a weighted sum of(i) a sum of absolute differences or a sum of values obtained throughtransform for a difference value by the transform unit (ii) the codeamount of the transform size information and the filter information) canbe used.

Once the evaluation values for all transform sizes are calculated, theboundary filtering unit 1305 determines, based on the evaluation values,the transform size of the transform unit and the filtering processmethod. The determined transform size is output to the filterinformation description unit 1313 along with the filter information andis described as the stream header information in a code sequence.

As described above, the partition method, the transform size, and thefiltering process method can be determined such that the differencebetween the input image and the filtered prediction image is smaller.

It should be noted that the boundary filtering unit 1305 calculates anevaluation value for each of all combinations of transform sizes.However, when the evaluation value is less than the predeterminedthreshold, the boundary filtering unit 1305 may cancel calculation ofthe subsequent evaluation values, and may determine that the filteringprocess method at that time is the optimal filtering process method.With this, the processing amount can be reduced.

Embodiment 6

FIG. 17 is a block diagram showing a functional configuration of animage decoding apparatus according to Embodiment 6 of the presentinvention. The image decoding apparatus 1700 decodes a code sequencegenerated by the image coding apparatus 1300 according to Embodiment 5of the present invention.

The image decoding apparatus 1700 includes a variable length decodingunit 1701, an inverse quantization unit 1702, an inverse transformationunit 1703, an addition unit 1704, a decoding control unit 1705, aboundary filtering unit 1706, a motion compensation unit 1707, areference picture memory 1708, and a filter information decoding unit1709. Here, the image decoding apparatus 1700 according to the presentembodiment is a configuration obtained by adding the filter informationdecoding unit 1709 to the image decoding apparatus 900 according toEmbodiment 2 of the present invention. The image decoding apparatus 1700performs an operation similar to the image decoding apparatus 900,except the operations of the decoding control unit 1705 and the boundaryfiltering unit 1706.

FIG. 18 is a flowchart showing a flow of filter information decodingprocessing according to Embodiment 6 of the present invention. Thefilter information decoding unit 1709 obtains a code sequence of filterinformation from the variable length decoding unit 1701, to decode thecode sequence (Step S1801). The filter information decoding unit 1709outputs, to the decoding control unit 1705, the filter boundarydetermination information (an offset value, the threshold, or the numberof boundary pixels for deriving a threshold for determining the filterboundary), the filtering process method, filter strength, or filtercoefficients that are obtained from decoding (Step S1802). It should benoted that the decoding control unit 1705 changes a method for obtainingan image from the motion compensation unit when the obtained informationfor determining the filter boundary is output to the boundary filteringunit 1706 and the filtering process method is expressed by Expression 2.With the above described operation, the code sequence generated by theimage decoding method according to Embodiment 5 can be accuratelydecoded.

Furthermore, a flow of operations in which the image coding methoddescribes the transform unit size in the code sequence will be shown inFIG. 19.

The filter information decoding unit 1709 obtains the filter informationand a code sequence of the transform size from the variable lengthdecoding unit 1701 to decode the transform unit size and the filterinformation (Step S1901). The filter information decoding unit 1709outputs the transform unit size obtained by decoding to the decodingcontrol unit 1705. The decoding control unit 1705 outputs the transformunit size to the inverse transformation unit 1703 (Step S1902).Moreover, the filter information decoding unit 1709 outputs, to thedecoding control unit 1705, the filter boundary determinationinformation and the filtering process method (coefficient or strength)that are obtained by decoding (S1903).

With this operation, even when the transform processing unit isdescribed in the code sequence, the decoding method according to thepresent invention makes it possible to accurately decode the codesequence.

Embodiment 7

In Embodiment 7 of the present invention, a method for describing thefilm header information FltStr indicating the information fordetermining the filter, the transform unit size for the filteringprocess, and the like, as the stream header information and for codingand decoding the film header information will be described.

FIG. 20 is a block diagram of a code sequence BS in the image codingmethod according to Embodiment 7 of the present invention. FIG. 20shows, in (a), a coded signal corresponding to a moving picture sequenceincluding at least a screen. The coded signal is composed of sequencedata SeqData that is data for all screens and sequence header SeqHdrthat is data common to all data for all screens.

The filter header information FltStr is information which includes theinformation for determining the filter, the transform unit size forfiltering process, and the like. For example, the filter headerinformation FltStr includes information for performing ON/OFF of filteron the target boundary within the transform unit (for example,NO_TU_MC_FLT_FLG=0 (OFF) and NO_TU_MC_FLT_FLG=1 (ON)). Moreover, whenthe filter is ON, the filter header information FltStr can include, asthe information for determining the filter, an OFFSET value for use incalculating a threshold for determining the filter, a threshold TH, orthe number of adjacent pixels from the target boundary. Furthermore, thefilter header information FltStr can include, as the information forfiltering process, the filtering process method, filter coefficients, orthe number of filter strength. Moreover, all of these do not have to beincluded in, and a predetermined value by the image coding method andthe image decoding method may be used for part of the information.Moreover, by switching between a table 2101 and a table 2102 with thetransform unit size shown in FIG. 21, a code number which represents thecombination of these pieces of information may be transmitted. It shouldbe noted that an overlap type of the filtering process method isprocessed by a formula given by Expression 2 and a filter tap type isprocessed by a formula given by Expression 4. Moreover, in an example ofFIG. 21, the transform unit size is used for switching between thetables. However, the switching is not limited to such.

The sequence header includes the filter header information FltStr.

In (b) of FIG. 20, the structure of the sequence data SeqData is shown.The sequence data SeqData includes a picture signal PicStr that is acoded signal of a picture corresponding to a screen.

In (c) of FIG. 20, the structure of the picture signal PicStr is shown.The picture signal PicStr is composed of the picture data PicData thatis data for a screen and the picture header PicHdr that is common to thewhole of the screen. The picture header PicHdr includes the filterheader information FltStr.

In (d) of FIG. 20, the structure of the picture data PicData is shown.The picture data PicData includes a slice signal SliceStr which is acoded signal of a slice composed of a group of block units.

In (e) of FIG. 20, the structure of the slice signal SliceStr is shown.The slice signal SliceStr is composed of a slice data SliceData that isdata for a slice and a slice header SliceHdr that is data common to alldata for a slice. By including the filter header information FltStr inthe slice header SliceHdr, a coded signal which has received on a slicedata SliceData basis can be accurately decoded.

It should be noted that when a plurality of picture signals PicStr areincluded in the sequence data SeqData, the filter header informationFltStr may be included only in some of the picture headers PicHdr ratherthan being included in all the picture headers PicHdr. Similarly, when aplurality of slice signals SliceStr are included in the picture dataPicData, the filter header information FltStr may be included in some ofthe slice headers SliceHdr rather than being included in all the sliceheaders SliceHdr. If the content of the filter header information iscommon to each of the slices and there is no filter header informationin the slice header as shown in (e) of FIG. 20, it is possible torestrict an increase in the number of bits by repeating the filterheader information FltStr by using, as a substitute, the filter headerinformation of another slice header.

Moreover, when the code sequence BS is transmitted not by a bit streamthat is consecutive but by a packet that is a unit of segmented data,the header part and the data part that is a part other than the headerpart may be separated and then transmitted separately. In that case, asshown in FIG. 20, it is impossible to be a bit stream that combines theheader part and the data part. However, in the case of packet, if anorder for transmitting the header part and the data part is notconsecutive, the header part corresponding to the data part istransmitted by a packet that is different from the data part. Even ifthey are not formed into a bit stream, the concept is the same as thebit stream described in FIG. 20.

Moreover, in the decoding method according to the present embodiment,the code sequence BS that is coded by the above described method isdecoded according to the following procedures. First, the filter datainformation FltStr included in the sequence header SeqHdr is obtainedand each of the pieces of information is held. Next, the filter headerinformation FltStr included in the picture header PicHdr is obtained andeach of the pieces of information is updated. Here, when the filterheader information FltStr does not exist or part of the filter headerinformation FltStr does not exist, the information included in thesequence header SeqHdr is held without any change. Similarly, the filterheader information FltStr included in the slice header SliceHdr isobtained and each of the pieces of information is updated.

With this, the code sequence can be accurately decoded.

Embodiment 8

In Embodiment 8 of the present invention, in the intra prediction image,when the boundary of the prediction image exists within the transformunit, an image coding method for performing filtering process on theboundary as similarly to the motion compensated image will be described.

FIG. 22 is a block diagram showing a functional configuration of animage coding apparatus according to Embodiment 8 of the presentinvention. As shown in FIG. 22, an image coding apparatus 2200 includesa difference unit 2201, a transformation and quantization unit 2202, aninverse quantization and inverse transformation unit 2203, an additionunit 2204, a coding distortion removal unit 2205, a reference picturememory 2206, an intra predicting unit 2207, a motion compensationpredicting unit 2208, a motion estimation unit 2209, a switch 2210, avariable length coding unit 2211, and a boundary filtering unit 2212.The reference picture memory 2206 may be constituted as an externalmemory connected to the image coding apparatus 2200.

An input image is input to the image coding apparatus 2200. In thereference picture memory 2206, a decoded image of the coded image(reconstructed image) is stored, and this is used as a reference picturewhen the input image is coded.

It should be noted that the difference unit 2201 operates similarly tothe difference unit 106, the transformation and quantization unit 2202operates similarly to the transformation unit 107 and the quantizationunit 108, the inverse quantization and inverse transformation unit 2203operates similarly to the inverse quantization unit 110 and the inversetransformation unit 111, the addition unit 2204 operates similarly tothe addition unit 112, the reference picture memory 2206 operatessimilarly to the reference picture memory 103, the boundary filteringunit 2212 operates similarly to the boundary filtering unit 105, and themotion estimation unit 2209 operates similarly to the motion estimationunit 102. Moreover, the motion compensation predicting unit 2208, theintra predicting unit 2207, and the switch 2210 operate similarly to thecoding control unit 104 when a motion compensated image is generated asthe prediction image MC.

The difference unit 2201 generates a difference image between aprediction image PR that is generated by a method to be described laterand an input image, and then outputs the generated difference image tothe transformation and quantization unit 2202. The transformation andquantization unit 2202 performs transformation and quantizationprocessing on the input difference image to generate a quantized signalQD, and then outputs the generated quantized signal QD to the inversequantization and inverse transformation unit 2203 and the variablelength coding unit 2211. The inverse quantization and inversetransformation unit 2203 performs inverse quantization and inversetransformation on the input quantized signal QD to generate a decodeddifference image, and then outputs the generated decoded differenceimage to the addition unit 2204. The addition unit 2204 adds the decodeddifference image obtained from the inverse quantization and inversetransformation unit 2203 and the prediction image PR to generate areconstructed image, and then outputs the generated reconstructed imageto the coding distortion removal unit 2205.

Here, the coding distortion removal unit 2205 performs, on the boundaryof the transform unit, coding distortion removal process (deblockingfilter) (for example, filtering process expressed by Expression 4 isperformed on the boundary of the transform unit). It should be notedthat the coding distortion removal unit 2205 may skip the process.

The generation of the prediction image MC is divided into motioncompensation prediction and inter-plane prediction and is switched bythe switch 2210.

The generation of the prediction image MC by intra prediction will bedescribed. The intra predicting unit 2207 obtains a reference picturefrom the reference picture memory 2206. The intra predicting unit 2207performs, based on the obtained reference picture, intra predictionaccording to a predetermined method (for example, by extending theadjacent pixel to a horizontal or vertical direction) to generate theintra prediction image of the target block, and then outputs the intraprediction image to “a” of the switch 2210. When “a” is selected by theswitch 2210, the intra prediction image is output to the boundaryfiltering unit 2212 as the prediction image MC. Furthermore, a methodfor partitioning the intra prediction is output to the boundaryfiltering unit 2212. When the boundary of the partitioned area of theintra prediction exists within the transform unit, the boundaryfiltering unit 2212 detects the boundary as the target boundary and thenperforms filtering process on the boundary.

The generation of the prediction image MC by motion compensationprediction will be described. The motion estimation unit 2209 estimates,from the reference picture, an area which corresponds to the predictionunit included in the target block. The motion estimation unit 2209outputs the motion vector MV indicating the position of the estimatedarea to the motion compensation predicting unit 2208 and the variablelength coding unit 2211. It should be noted that a weighted sum of (i) asum of difference between the input image and the estimating targetimage in the reference picture and (ii) the motion vector can be usedfor a method for estimating an area. In this case, the motion estimationunit 2209 may search a position of the area which takes the minimumvalue in a search range of the predetermined reference picture (forexample, in a unit of horizontal 32 pixels and vertical 16 pixels) andoutput the motion vector indicating the position of the searched area.

The motion compensation predicting unit 2208 picks out, for eachprediction unit, an image in the reference picture indicated by theobtained motion vector MV and then outputs the image as the motioncompensated image of the target block to “b” of the switch 2210.

When “b” is selected by the switch 2210, the motion compensated image isoutput to the boundary filtering unit 2212 as the prediction image MC.Furthermore, a method for partitioning the motion compensated imageoutput to the boundary filtering unit 2212. In this case, the boundaryfiltering unit 2212 operates as similar to that in other embodiments.

The switch 2210 switches, according to a predetermined method, betweenoutputting an intra prediction image as the prediction image MC andoutputting a motion compensated image as the prediction image MC. Inother words, the switch 2210 selects one of the intra prediction imageand the motion compensated image as the prediction image MC. Forexample, the switch 2210 may compare the sum of the difference betweenthe input image and the intra prediction image with the sum of thedifference between the input image and the motion compensated image, andmay output the image having a smaller sum of difference as theprediction image MC. The information about a method for generating theprediction image PR is output to the variable length coding unit 2211 asprediction image generation information CI.

It should be noted that when the filter information determined by theboundary filtering unit 2212 is added to a code sequence, the boundaryfiltering unit 2212 may output, to the variable length coding unit 2211,the filter information along with the prediction image generationinformation CI.

The variable length coding unit 2211 performs variable length coding onthe quantized signal QD, the motion vector MV, and the prediction imagegeneration information CI, and then output them as the coded stream BS.

By taking such a configuration, it is possible to perform filteringprocess on a part of the boundary of the prediction unit located withinthe transform unit described in Embodiment 1 in the intra prediction.With this, at a time of intra prediction, a difference signal can besmaller and a code amount can be reduced.

It should be noted that the image coding apparatus 2200 includes boththe intra predicting unit 2207 and the motion compensation predictingunit 2208. But the image coding apparatus 2200 may include only theintra predicting unit 2207. In this case, the motion estimation unit2209 and the switch 2210 are not necessary.

Embodiment 9

In Embodiment 9 of the present invention, in the intra prediction image,when the boundary of the prediction image exists within the transformunit, an image decoding method for performing filtering process on theboundary as similarly to the motion compensated image will be described.

FIG. 23 is a block diagram showing a functional configuration of animage decoding apparatus according to Embodiment 9 of the presentinvention. As shown in FIG. 23, the image decoding apparatus 2300includes a variable length decoding unit 2301, an inverse quantizationand inverse transformation unit 2302, and an addition unit 2303, acoding distortion removal unit 2304, a reference picture memory 2305, anintra predicting unit 2306, a motion compensation predicting unit 2307,a switch 2308, and a boundary filtering unit 2309. The reference picturememory 2305 may be constituted as an external memory connected to theimage decoding apparatus 2300.

Here, in the reference picture memory 2305, a decoded image is alreadystored, and this is used as a reference picture when the input image isdecoded.

It should be noted that the inverse quantization and inversetransformation unit 2302 operates similarly to the inverse quantizationunit 902 and the inverse transformation unit 903, the addition unit 2303operates similarly to the addition unit 904, the reference picturememory 2305 operates similarly to the reference picture memory 908, andthe boundary filtering unit 2309 operates similarly to the boundaryfiltering unit 906. The motion compensation unit 907 is implemented asthe motion compensation predicting unit 2307 and the reference picturememory 2305, the decoding control unit 905 is implemented as the motioncompensation predicting unit 2307, the intra predicting unit 2306, andthe switch 2308.

A code sequence BS is input to the image decoding apparatus 2300. Thevariable length decoding unit 2301 performs variable length decoding onthe code sequence BS and then obtains the prediction image generationimage CI, the motion information MV, and the quantized signal QD. Thevariable length decoding unit 2301 outputs the prediction imagegeneration information CI to the switch 2308 and the boundary filteringunit 2309, outputs the motion information MV to the motion compensationpredicting unit 2307, and outputs the quantized signal QD to the inversequantization and inverse transformation unit 2302.

The inverse quantization and inverse transformation unit 2302 performsinverse quantization and inverse transformation on the quantized signalQD to generate a decoded difference image, and then outputs the decodeddifference image to the addition unit 2303. The addition unit 2303 addsthe decoded difference image and the prediction image PR to be describedlater and then outputs the added result to the coding distortion removalunit 2304.

The coding distortion removal unit 2304 performs coding distortionremoval on the input signal and then outputs it as the decoded imagewith respect to the reference picture memory 2305.

Here, the coding distortion removal unit 2304 performs, on the boundaryof the transform unit, coding distortion removal (deblocking filter)(for example, filtering process expressed by Expression 4 is performedon the boundary of the transform unit), and may skip the codingdistortion removal process.

It should be noted that the method for generating the prediction image,based on the prediction image generation information CI, similarly tothe image coding apparatus described in Embodiment 8, based on thereference picture obtained from the reference picture memory 2305, isgenerated by the intra predicting unit 2306, or the motion informationMV and the motion compensation predicting unit 2307. The generated imageis output to the boundary filtering unit 2309, and filtering process isperformed on the prediction image boundary within the transformprocessing unit. It should be noted that when the filter information iscoded with respect to a code sequence, decoding is performed by thevariable length decoding unit 2301, input is performed to the boundaryfiltering unit 2309 along with the prediction image generationinformation CI, and filtering process is performed based on the filterinformation.

With this configuration, a code sequence in which the filtering processis performed also with respect to the intra prediction on the predictionboundary within the transform processing unit described in Embodiment 8can be accurately decoded.

Embodiment 10

The processing described in each of embodiments can be simplyimplemented in an independent computer system, by recording, in arecording medium, a program for implementing the configurations of themoving picture coding method (image coding method) and the movingpicture decoding method (image decoding method) described in each ofembodiments. The recording media may be any recording media as long asthe program can be recorded, such as a magnetic disk, an optical disk, amagnetic optical disk, an IC card, and a semiconductor memory.

Hereinafter, the applications to the moving picture coding method andthe moving picture decoding method described in each of embodiments andsystems using thereof will be described.

FIG. 24 illustrates an overall configuration of a content providingsystem ex100 for implementing content distribution services. The areafor providing communication services is divided into cells of desiredsize, and base stations ex106, ex107, ex108, ex109, and ex110 which arefixed wireless stations are placed in each of the cells.

The content providing system ex100 is connected to devices, such as acomputer ex111, a personal digital assistant (PDA) ex112, a cameraex113, a cellular phone ex114 and a game machine ex115, via the Internetex101, an Internet service provider ex102, a telephone network ex104, aswell as the base stations ex106 to ex110, respectively.

However, the configuration of the content providing system ex100 is notlimited to the configuration shown in FIG. 24, and a combination inwhich any of the elements are connected is acceptable. In addition, eachdevice may be directly connected to the telephone network ex104, ratherthan via the base stations ex106 to ex110 which are the fixed wirelessstations. Furthermore, the devices may be interconnected to each othervia a short distance wireless communication and others.

The camera ex113, such as a digital video camera, is capable ofcapturing video. A camera ex116, such as a digital camera, is capable ofcapturing both still images and video. Furthermore, the cellular phoneex114 may be the one that meets any of the standards such as GlobalSystem for Mobile Communications (GSM) (registered trademark), CodeDivision Multiple Access (CDMA), Wideband-Code Division Multiple Access(W-CDMA), Long Term Evolution (LTE), and High Speed Packet Access(HSPA). Alternatively, the cellular phone ex114 may be a PersonalHandyphone System (PHS).

In the content providing system ex100, a streaming server ex103 isconnected to the camera ex113 and others via the telephone network ex104and the base station ex109, which enables distribution of images of alive show and others. In such a distribution, a content (for example,video of a music live show) captured by the user using the camera ex113is coded as described above in each of embodiments (i.e., the camerafunctions as the image coding apparatus according to an aspect of thepresent invention), and the coded content is transmitted to thestreaming server ex103. On the other hand, the streaming server ex103carries out stream distribution of the transmitted content data to theclients upon their requests. The clients include the computer ex111, thePDA ex112, the camera ex113, the cellular phone ex114, and the gamemachine ex115 that are capable of decoding the above-mentioned codeddata. Each of the devices that have received the distributed datadecodes and reproduces the coded data.

The captured data may be coded by the camera ex113 or the streamingserver ex103 that transmits the data, or the coding processes may beshared between the camera ex113 and the streaming server ex103.Similarly, the distributed data may be decoded by the clients or thestreaming server ex103, or the decoding processes may be shared betweenthe clients and the streaming server ex103. Furthermore, the data of thestill images and video captured by not only the camera ex113 but alsothe camera ex116 may be transmitted to the streaming server ex103through the computer ex111. The coding processes may be performed by thecamera ex116, the computer ex111, or the streaming server ex103, orshared among them.

Furthermore, the coding and decoding processes may be performed by anLSI ex500 generally included in each of the computer ex111 and thedevices. The LSI ex500 may be configured of a single chip or a pluralityof chips. Software for coding and decoding video may be integrated intosome type of a recording medium (such as a CD-ROM, a flexible disk, anda hard disk) that is readable by the computer ex111 and others, and thecoding and decoding processes may be performed using the software.Furthermore, when the cellular phone ex114 is equipped with a camera,the video data obtained by the camera may be transmitted. The video datais data coded by the LSI ex500 included in the cellular phone ex114.

Furthermore, the streaming server ex103 may be composed of servers andcomputers, and may decentralize data and process the decentralized data,record, or distribute data.

As described above, the clients may receive and reproduce the coded datain the content providing system ex100. In other words, the clients canreceive and decode information transmitted by the user, and reproducethe decoded data in real time in the content providing system ex100, sothat the user who does not have any particular right and equipment canimplement personal broadcasting.

Aside from the example of the content providing system ex100, at leastone of the moving picture coding apparatus (image coding apparatus) andthe moving picture decoding apparatus (image decoding apparatus)described in each of embodiments may be implemented in a digitalbroadcasting system ex200 illustrated in FIG. 25. More specifically, abroadcast station ex201 communicates or transmits, via radio waves to abroadcast satellite ex202, multiplexed data obtained by multiplexingaudio data and others onto video data. The video data is data coded bythe moving picture coding method described in each of embodiments. Uponreceipt of the multiplexed data, the broadcast satellite ex202 transmitsradio waves for broadcasting. Then, a home-use antenna ex204 with asatellite broadcast reception function receives the radio waves. Next, adevice such as a television (receiver) ex300 and a set top box (STB)ex217 decodes the received multiplexed data, and reproduces the decodeddata.

Furthermore, a reader/recorder ex218 (i) reads and decodes themultiplexed data recorded on a recording medium ex215, such as a DVD anda BD, or (ii) codes video signals in the recording medium ex215, and insome cases, writes data obtained by multiplexing an audio signal on thecoded data. The reader/recorder ex218 can include the moving picturedecoding apparatus or the moving picture coding apparatus as shown ineach of embodiments. In this case, the reproduced video signals aredisplayed on the monitor ex219, and can be reproduced by another deviceor system using the recording medium ex215 on which the multiplexed datais recorded. It is also possible to implement the moving picturedecoding apparatus in the set top box ex217 connected to the cable ex203for a cable television or to the antenna ex204 for satellite and/orterrestrial broadcasting, so as to display the video signals on themonitor ex219 of the television ex300. The moving picture decodingapparatus may be implemented not in the set top box but in thetelevision ex300.

FIG. 26 illustrates the television (receiver) ex300 that uses the movingpicture coding method and the moving picture decoding method describedin each of embodiments. The television ex300 includes: a tuner ex301that obtains or provides multiplexed data obtained by multiplexing audiodata onto video data, through the antenna ex204 or the cable ex203, etc.that receives a broadcast; a modulation/demodulation unit ex302 thatdemodulates the received multiplexed data or modulates data intomultiplexed data to be supplied outside; and amultiplexing/demultiplexing unit ex303 that demultiplexes the modulatedmultiplexed data into video data and audio data, or multiplexes videodata and audio data coded by a signal processing unit ex306 into data.

The television ex300 further includes: a signal processing unit ex306including an audio signal processing unit ex304 and a video signalprocessing unit ex305 that decode audio data and video data and codeaudio data and video data, respectively; and an output unit ex309including a speaker ex307 that provides the decoded audio signal, and adisplay unit ex308 that displays the decoded video signal, such as adisplay. Furthermore, the television ex300 includes an interface unitex317 including an operation input unit ex312 that receives an input ofa user operation. Furthermore, the television ex300 includes a controlunit ex310 that controls overall each constituent element of thetelevision ex300, and a power supply circuit unit ex311 that suppliespower to each of the elements. Other than the operation input unitex312, the interface unit ex317 may include: a bridge ex313 that isconnected to an external device, such as the reader/recorder ex218; aslot unit ex314 for enabling attachment of the recording medium ex216,such as an SD card; a driver ex315 to be connected to an externalrecording medium, such as a hard disk; and a modem ex316 to be connectedto a telephone network. Here, the recording medium ex216 canelectrically record information using a non-volatile/volatilesemiconductor memory element for storage. The constituent elements ofthe television ex300 are connected to each other through a synchronousbus.

First, the configuration in which the television ex300 decodesmultiplexed data obtained from outside through the antenna ex204 andothers and reproduces the decoded data will be described. In thetelevision ex300, upon a user operation through a remote controllerex220 and others, the multiplexing/demultiplexing unit ex303demultiplexes the multiplexed data demodulated by themodulation/demodulation unit ex302, under control of the control unitex310 including a CPU. Furthermore, the audio signal processing unitex304 decodes the demultiplexed audio data, and the video signalprocessing unit ex305 decodes the demultiplexed video data, using thedecoding method described in each of embodiments, in the televisionex300. The output unit ex309 provides the decoded video signal and audiosignal outside, respectively. When the output unit ex309 provides thevideo signal and the audio signal, the signals may be temporarily storedin buffers ex318 and ex319, and others so that the signals arereproduced in synchronization with each other. Furthermore, thetelevision ex300 may read multiplexed data not through a broadcast andothers but from the recording media ex215 and ex216, such as a magneticdisk, an optical disk, and a SD card. Next, a configuration in which thetelevision ex300 codes an audio signal and a video signal, and transmitsthe data outside or writes the data on a recording medium will bedescribed. In the television ex300, upon a user operation through theremote controller ex220 and others, the audio signal processing unitex304 codes an audio signal, and the video signal processing unit ex305codes a video signal, under control of the control unit ex310 using thecoding method described in each of embodiments. Themultiplexing/demultiplexing unit ex303 multiplexes the coded videosignal and audio signal, and provides the resulting signal outside. Whenthe multiplexing/demultiplexing unit ex303 multiplexes the video signaland the audio signal, the signals may be temporarily stored in thebuffers ex320 and ex321, and others so that the signals are reproducedin synchronization with each other. Here, the buffers ex318, ex319,ex320, and ex321 may be plural as illustrated, or at least one buffermay be shared in the television ex300. Furthermore, data may be storedin a buffer so that the system overflow and underflow may be avoidedbetween the modulation/demodulation unit ex302 and themultiplexing/demultiplexing unit ex303, for example.

Furthermore, the television ex300 may include a configuration forreceiving an AV input from a microphone or a camera other than theconfiguration for obtaining audio and video data from a broadcast or arecording medium, and may code the obtained data. Although thetelevision ex300 can code, multiplex, and provide outside data in thedescription, it may be capable of only receiving, decoding, andproviding outside data but not the coding, multiplexing, and providingoutside data.

Furthermore, when the reader/recorder ex218 reads or writes multiplexeddata from or on a recording medium, one of the television ex300 and thereader/recorder ex218 may decode or code the multiplexed data, and thetelevision ex300 and the reader/recorder ex218 may share the decoding orcoding.

As an example, FIG. 27 illustrates a configuration of an informationreproducing/recording unit ex400 when data is read or written from or onan optical disk. The information reproducing/recording unit ex400includes constituent elements ex401, ex402, ex403, ex404, ex405, ex406,and ex407 to be described hereinafter. The optical head ex401 irradiatesa laser spot in a recording surface of the recording medium ex215 thatis an optical disk to write information, and detects reflected lightfrom the recording surface of the recording medium ex215 to read theinformation. The modulation recording unit ex402 electrically drives asemiconductor laser included in the optical head ex401, and modulatesthe laser light according to recorded data. The reproductiondemodulating unit ex403 amplifies a reproduction signal obtained byelectrically detecting the reflected light from the recording surfaceusing a photo detector included in the optical head ex401, anddemodulates the reproduction signal by separating a signal componentrecorded on the recording medium ex215 to reproduce the necessaryinformation. The buffer ex404 temporarily holds the information to berecorded on the recording medium ex215 and the information reproducedfrom the recording medium ex215. The disk motor ex405 rotates therecording medium ex215. The servo control unit ex406 moves the opticalhead ex401 to a predetermined information track while controlling therotation drive of the disk motor ex405 so as to follow the laser spot.The system control unit ex407 controls overall the informationreproducing/recording unit ex400. The reading and writing processes canbe implemented by the system control unit ex407 using variousinformation stored in the buffer ex404 and generating and adding newinformation as necessary, and by the modulation recording unit ex402,the reproduction demodulating unit ex403, and the servo control unitex406 that record and reproduce information through the optical headex401 while being operated in a coordinated manner. The system controlunit ex407 includes, for example, a microprocessor, and executesprocessing by causing a computer to execute a program for read andwrite.

Although the optical head ex401 irradiates a laser spot in thedescription, it may perform high-density recording using near fieldlight.

FIG. 28 illustrates the recording medium ex215 that is the optical disk.On the recording surface of the recording medium ex215, guide groovesare spirally formed, and an information track ex230 records, in advance,address information indicating an absolute position on the diskaccording to change in a shape of the guide grooves. The addressinformation includes information for determining positions of recordingblocks ex231 that are a unit for recording data. Reproducing theinformation track ex230 and reading the address information in anapparatus that records and reproduces data can lead to determination ofthe positions of the recording blocks. Furthermore, the recording mediumex215 includes a data recording area ex233, an inner circumference areaex232, and an outer circumference area ex234. The data recording areaex233 is an area for use in recording the user data. The innercircumference area ex232 and the outer circumference area ex234 that areinside and outside of the data recording area ex233, respectively arefor specific use except for recording the user data. The informationreproducing/recording unit 400 reads and writes coded audio, coded videodata, or multiplexed data obtained by multiplexing the coded audio andvideo data, from and on the data recording area ex233 of the recordingmedium ex215.

Although an optical disk having a layer, such as a DVD and a BD isdescribed as an example in the description, the optical disk is notlimited to such, and may be an optical disk having a multilayerstructure and capable of being recorded on a part other than thesurface. Furthermore, the optical disk may have a structure formultidimensional recording/reproduction, such as recording ofinformation using light of colors with different wavelengths in the sameportion of the optical disk and for recording information havingdifferent layers from various angles.

Furthermore, a car ex210 having an antenna ex205 can receive data fromthe satellite ex202 and others, and reproduce video on a display devicesuch as a car navigation system ex211 set in the car ex210, in thedigital broadcasting system ex200. Here, a configuration of the carnavigation system ex211 will be a configuration, for example, includinga GPS receiving unit from the configuration illustrated in FIG. 26. Thesame will be true for the configuration of the computer ex111, thecellular phone ex114, and others. Furthermore, similarly to thetelevision ex300, a terminal such as the cellular phone ex114 probablyhave 3 types of implementation configurations including not only (i) atransmitting and receiving terminal including both a coding apparatusand a decoding apparatus, but also (ii) a transmitting terminalincluding only a coding apparatus and (iii) a receiving terminalincluding only a decoding apparatus. Although the digital broadcastingsystem ex200 receives and transmits the multiplexed data obtained bymultiplexing audio data onto video data in the description, themultiplexed data may be data obtained by multiplexing not audio data butcharacter data related to video onto video data, and may be notmultiplexed data but video data itself.

As such, the moving picture coding method and the moving picturedecoding method in each of embodiments can be used in any of the devicesand systems described. Thus, the advantages described in each ofembodiments can be obtained.

Furthermore, the present invention is not limited to embodiments, andvarious modifications and revisions are possible without departing fromthe scope of the present invention.

Embodiment 11

Video data can be generated by switching, as necessary, between (i) themoving picture coding method or the moving picture coding apparatusshown in each of embodiments and (ii) a moving picture coding method ora moving picture coding apparatus in conformity with a differentstandard, such as MPEG-2, MPEG-4 AVC, and VC-1.

Here, when a plurality of video data that conforms to the differentstandards is generated and is then decoded, the decoding methods need tobe selected to conform to the different standards. However, since towhich standard each of the plurality of the video data to be decodedconform cannot be detected, there is a problem that an appropriatedecoding method cannot be selected.

In order to solve the problem, multiplexed data obtained by multiplexingaudio data and others onto video data has a structure includingidentification information indicating to which standard the video dataconforms. The specific structure of the multiplexed data including thevideo data generated in the moving picture coding method and by themoving picture coding apparatus shown in each of embodiments will behereinafter described. The multiplexed data is a digital stream in theMPEG-2 Transport Stream format.

FIG. 29 illustrates a structure of the multiplexed data. As illustratedin FIG. 29, the multiplexed data can be obtained by multiplexing atleast one of a video stream, an audio stream, a presentation graphicsstream (PG), and an interactive graphics stream. The video streamrepresents primary video and secondary video of a movie, the audiostream (IG) represents a primary audio part and a secondary audio partto be mixed with the primary audio part, and the presentation graphicsstream represents subtitles of the movie. Here, the primary video isnormal video to be displayed on a screen, and the secondary video isvideo to be displayed on a smaller window in the primary video.Furthermore, the interactive graphics stream represents an interactivescreen to be generated by arranging the GUI components on a screen. Thevideo stream is coded in the moving picture coding method or by themoving picture coding apparatus shown in each of embodiments, or in amoving picture coding method or by a moving picture coding apparatus inconformity with a conventional standard, such as MPEG-2, MPEG-4 AVC, andVC-1. The audio stream is coded in accordance with a standard, such asDolby-AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD, and linear PCM.

Each stream included in the multiplexed data is identified by PID. Forexample, 0x1011 is allocated to the video stream to be used for video ofa movie, 0x1100 to 0x111F are allocated to the audio streams, 0x1200 to0x121F are allocated to the presentation graphics streams, 0x1400 to0x141F are allocated to the interactive graphics streams, 0x1B00 to0x1B1F are allocated to the video streams to be used for secondary videoof the movie, and 0x1A00 to 0x1A1F are allocated to the audio streams tobe used for the secondary audio to be mixed with the primary audio.

FIG. 30 schematically illustrates how data is multiplexed. First, avideo stream ex235 composed of video frames and an audio stream ex238composed of audio frames are transformed into a stream of PES packetsex236 and a stream of PES packets ex239, and further into TS packetsex237 and TS packets ex240, respectively. Similarly, data of apresentation graphics stream ex241 and data of an interactive graphicsstream ex244 are transformed into a stream of PES packets ex242 and astream of PES packets ex245, and further into TS packets ex243 and TSpackets ex246, respectively. These TS packets are multiplexed into astream to obtain multiplexed data ex247.

FIG. 31 illustrates how a video stream is stored in a stream of PESpackets in more detail. The first bar in FIG. 17 shows a video framestream in a video stream. The second bar shows the stream of PESpackets. As indicated by arrows denoted as yy1, yy2, yy3, and yy4 inFIG. 17, the video stream is divided into pictures as I pictures, Bpictures, and P pictures each of which is a video presentation unit, andthe pictures are stored in a payload of each of the PES packets. Each ofthe PES packets has a PES header, and the PES header stores aPresentation Time-Stamp (PTS) indicating a display time of the picture,and a Decoding Time-Stamp (DTS) indicating a decoding time of thepicture.

FIG. 32 illustrates a format of TS packets to be finally written on themultiplexed data. Each of the TS packets is a 188-byte fixed lengthpacket including a 4-byte TS header having information, such as a PIDfor identifying a stream and a 184-byte TS payload for storing data. ThePES packets are divided, and stored in the TS payloads, respectively.When a BD ROM is used, each of the TS packets is given a 4-byteTP_Extra_Header, thus resulting in 192-byte source packets. The sourcepackets are written on the multiplexed data. The TP_Extra_Header storesinformation such as an Arrival_Time_Stamp (ATS). The ATS shows atransfer start time at which each of the TS packets is to be transferredto a PID filter. The source packets are arranged in the multiplexed dataas shown at the bottom of FIG. 32. The numbers incrementing from thehead of the multiplexed data are called source packet numbers (SPNs).

Each of the TS packets included in the multiplexed data includes notonly streams of audio, video, subtitles and others, but also a ProgramAssociation Table (PAT), a Program Map Table (PMT), and a Program ClockReference (PCR). The PAT shows what a PID in a PMT used in themultiplexed data indicates, and a PID of the PAT itself is registered aszero. The PMT stores PIDs of the streams of video, audio, subtitles andothers included in the multiplexed data, and attribute information ofthe streams corresponding to the PIDs. The PMT also has variousdescriptors relating to the multiplexed data. The descriptors haveinformation such as copy control information showing whether copying ofthe multiplexed data is permitted or not. The PCR stores STC timeinformation corresponding to an ATS showing when the PCR packet istransferred to a decoder, in order to achieve synchronization between anArrival Time Clock (ATC) that is a time axis of ATSs, and an System TimeClock (STC) that is a time axis of PTSs and DTSs.

FIG. 33 illustrates the data structure of the PMT in detail. A PMTheader is disposed at the top of the PMT. The PMT header describes thelength of data included in the PMT and others. A plurality ofdescriptors relating to the multiplexed data is disposed after the PMTheader. Information such as the copy control information is described inthe descriptors. After the descriptors, a plurality of pieces of streaminformation relating to the streams included in the multiplexed data isdisposed. Each piece of stream information includes stream descriptorseach describing information, such as a stream type for identifying acompression codec of a stream, a stream PID, and stream attributeinformation (such as a frame rate or an aspect ratio). The streamdescriptors are equal in number to the number of streams in themultiplexed data.

When the multiplexed data is recorded on a recording medium and others,it is recorded together with multiplexed data information files.

Each of the multiplexed data information files is management informationof the multiplexed data as shown in FIG. 34. The multiplexed datainformation files are in one to one correspondence with the multiplexeddata, and each of the files includes multiplexed data information,stream attribute information, and an entry map.

As illustrated in FIG. 34, the multiplexed data information includes asystem rate, a reproduction start time, and a reproduction end time. Thesystem rate indicates the maximum transfer rate at which a system targetdecoder to be described later transfers the multiplexed data to a PIDfilter. The intervals of the ATSs included in the multiplexed data areset to not higher than a system rate. The reproduction start timeindicates a PTS in a video frame at the head of the multiplexed data. Aninterval of one frame is added to a PTS in a video frame at the end ofthe multiplexed data, and the PTS is set to the reproduction end time.

As shown in FIG. 35, a piece of attribute information is registered inthe stream attribute information, for each PID of each stream includedin the multiplexed data. Each piece of attribute information hasdifferent information depending on whether the corresponding stream is avideo stream, an audio stream, a presentation graphics stream, or aninteractive graphics stream. Each piece of video stream attributeinformation carries information including what kind of compression codecis used for compressing the video stream, and the resolution, aspectratio and frame rate of the pieces of picture data that is included inthe video stream. Each piece of audio stream attribute informationcarries information including what kind of compression codec is used forcompressing the audio stream, how many channels are included in theaudio stream, which language the audio stream supports, and how high thesampling frequency is. The video stream attribute information and theaudio stream attribute information are used for initialization of adecoder before the player plays back the information.

In the present embodiment, the multiplexed data to be used is of astream type included in the PMT. Furthermore, when the multiplexed datais recorded on a recording medium, the video stream attributeinformation included in the multiplexed data information is used. Morespecifically, the moving picture coding method or the moving picturecoding apparatus described in each of embodiments includes a step or aunit for allocating unique information indicating video data generatedby the moving picture coding method or the moving picture codingapparatus in each of embodiments, to the stream type included in the PMTor the video stream attribute information. With the configuration, thevideo data generated by the moving picture coding method or the movingpicture coding apparatus described in each of embodiments can bedistinguished from video data that conforms to another standard.

Furthermore, FIG. 36 illustrates steps of the moving picture decodingmethod according to the present embodiment. In Step exS100, the streamtype included in the PMT or the video stream attribute informationincluded in the multiplexed data information is obtained from themultiplexed data. Next, in Step exS101, it is determined whether or notthe stream type or the video stream attribute information indicates thatthe multiplexed data is generated by the moving picture coding method orthe moving picture coding apparatus in each of embodiments. When it isdetermined that the stream type or the video stream attributeinformation indicates that the multiplexed data is generated by themoving picture coding method or the moving picture coding apparatus ineach of embodiments, in Step exS102, decoding is performed by the movingpicture decoding method in each of embodiments. Furthermore, when thestream type or the video stream attribute information indicatesconformance to the conventional standards, such as MPEG-2, MPEG-4 AVC,and VC-1, in Step exS103, decoding is performed by a moving picturedecoding method in conformity with the conventional standards.

As such, allocating a new unique value to the stream type or the videostream attribute information enables determination whether or not themoving picture decoding method or the moving picture decoding apparatusthat is described in each of embodiments can perform decoding. Even whenmultiplexed data that conforms to a different standard is input, anappropriate decoding method or apparatus can be selected. Thus, itbecomes possible to decode information without any error. Furthermore,the moving picture coding method or apparatus, or the moving picturedecoding method or apparatus in the present embodiment can be used inthe devices and systems described above.

Embodiment 12

Each of the moving picture coding method, the moving picture codingapparatus, the moving picture decoding method, and the moving picturedecoding apparatus in each of embodiments is typically achieved in theform of an integrated circuit or a Large Scale Integrated (LSI) circuit.As an example of the LSI, FIG. 37 illustrates a configuration of the LSIex500 that is made into one chip. The LSI ex500 includes elements ex501,ex502, ex503, ex504, ex505, ex506, ex507, ex508, and ex509 to bedescribed below, and the elements are connected to each other through abus ex510. The power supply circuit unit ex505 is activated by supplyingeach of the elements with power when the power supply circuit unit ex505is turned on.

For example, when coding is performed, the LSI ex500 receives an AVsignal from a microphone ex117, a camera ex113, and others through an AVIO ex509 under control of a control unit ex501 including a CPU ex502, amemory controller ex503, a stream controller ex504, and a drivingfrequency control unit ex512. The received AV signal is temporarilystored in an external memory ex511, such as an SDRAM. Under control ofthe control unit ex501, the stored data is segmented into data portionsaccording to the processing amount and speed to be transmitted to asignal processing unit ex507. Then, the signal processing unit ex507codes an audio signal and/or a video signal. Here, the coding of thevideo signal is the coding described in each of embodiments.Furthermore, the signal processing unit ex507 sometimes multiplexes thecoded audio data and the coded video data, and a stream IO ex506provides the multiplexed data outside. The provided multiplexed data istransmitted to the base station ex107, or written on the recordingmedium ex215. When data sets are multiplexed, the data should betemporarily stored in the buffer ex508 so that the data sets aresynchronized with each other.

Although the memory ex511 is an element outside the LSI ex500, it may beincluded in the LSI ex500. The buffer ex508 is not limited to onebuffer, but may be composed of buffers. Furthermore, the LSI ex500 maybe made into one chip or a plurality of chips.

Furthermore, although the control unit ex501 includes the CPU ex502, thememory controller ex503, the stream controller ex504, the drivingfrequency control unit ex512, the configuration of the control unitex501 is not limited to such. For example, the signal processing unitex507 may further include a CPU. Inclusion of another CPU in the signalprocessing unit ex507 can increase the processing speed. Furthermore, asanother example, the CPU ex502 may serve as or be a part of the signalprocessing unit ex507, and, for example, may include an audio signalprocessing unit. In such a case, the control unit ex501 includes thesignal processing unit ex507 or the CPU ex502 including a part of thesignal processing unit ex507.

The name used here is LSI, but it may also be called IC, system LSI,super LSI, or ultra LSI depending on the degree of integration.

Moreover, ways to achieve integration are not limited to the LSI, and aspecial circuit or a general purpose processor and so forth can alsoachieve the integration. Field Programmable Gate Array (FPGA) that canbe programmed after manufacturing LSIs or a reconfigurable processorthat allows re-configuration of the connection or configuration of anLSI can be used for the same purpose.

In the future, with advancement in semiconductor technology, a brand-newtechnology may replace LSI. The functional blocks can be integratedusing such a technology. The possibility is that the present inventionis applied to biotechnology.

Embodiment 13

When video data generated in the moving picture coding method or by themoving picture coding apparatus described in each of embodiments isdecoded, compared to when video data that conforms to a conventionalstandard, such as MPEG-2, MPEG-4 AVC, and VC-1 is decoded, theprocessing amount probably increases. Thus, the LSI ex500 needs to beset to a driving frequency higher than that of the CPU ex502 to be usedwhen video data in conformity with the conventional standard is decoded.However, when the driving frequency is set higher, there is a problemthat the power consumption increases.

In order to solve the problem, the moving picture decoding apparatus,such as the television ex300 and the LSI ex500 is configured todetermine to which standard the video data conforms, and switch betweenthe driving frequencies according to the determined standard. FIG. 38illustrates a configuration ex800 in the present embodiment. A drivingfrequency switching unit ex803 sets a driving frequency to a higherdriving frequency when video data is generated by the moving picturecoding method or the moving picture coding apparatus described in eachof embodiments. Then, the driving frequency switching unit ex803instructs a decoding processing unit ex801 that executes the movingpicture decoding method described in each of embodiments to decode thevideo data. When the video data conforms to the conventional standard,the driving frequency switching unit ex803 sets a driving frequency to alower driving frequency than that of the video data generated by themoving picture coding method or the moving picture coding apparatusdescribed in each of embodiments. Then, the driving frequency switchingunit ex803 instructs the decoding processing unit ex802 that conforms tothe conventional standard to decode the video data.

More specifically, the driving frequency switching unit ex803 includesthe CPU ex502 and the driving frequency control unit ex512 in FIG. 37.Here, each of the decoding processing unit ex801 that executes themoving picture decoding method described in each of embodiments and thedecoding processing unit ex802 that conforms to the conventionalstandard corresponds to the signal processing unit ex507 in FIG. 37. TheCPU ex502 determines to which standard the video data conforms. Then,the driving frequency control unit ex512 determines a driving frequencybased on a signal from the CPU ex502. Furthermore, the signal processingunit ex507 decodes the video data based on the signal from the CPUex502. For example, the identification information described inEmbodiment B is probably used for identifying the video data. Theidentification information is not limited to the one described inEmbodiment 11 but may be any information as long as the informationindicates to which standard the video data conforms. For example, whenwhich standard video data conforms to can be determined based on anexternal signal for determining that the video data is used for atelevision or a disk, etc., the determination may be made based on suchan external signal. Furthermore, the CPU ex502 selects a drivingfrequency based on, for example, a look-up table in which the standardsof the video data are associated with the driving frequencies as shownin FIG. 40. The driving frequency can be selected by storing the look-uptable in the buffer ex508 and in an internal memory of an LSI, and withreference to the look-up table by the CPU ex502.

FIG. 39 illustrates steps for executing a method in the presentembodiment. First, in Step exS200, the signal processing unit ex507obtains identification information from the multiplexed data. Next, inStep exS201, the CPU ex502 determines whether or not the video data isgenerated by the coding method and the coding apparatus described ineach of embodiments, based on the identification information. When thevideo data is generated by the moving picture coding method and themoving picture coding apparatus described in each of embodiments, inStep exS202, the CPU ex502 transmits a signal for setting the drivingfrequency to a higher driving frequency to the driving frequency controlunit ex512. Then, the driving frequency control unit ex512 sets thedriving frequency to the higher driving frequency. On the other hand,when the identification information indicates that the video dataconforms to the conventional standard, such as MPEG-2, MPEG-4 AVC, andVC-1, in Step exS203, the CPU ex502 transmits a signal for setting thedriving frequency to a lower driving frequency to the driving frequencycontrol unit ex512. Then, the driving frequency control unit ex512 setsthe driving frequency to the lower driving frequency than that in thecase where the video data is generated by the moving picture codingmethod and the moving picture coding apparatus described in each ofembodiment.

Furthermore, along with the switching of the driving frequencies, thepower conservation effect can be increased by changing the voltage to beapplied to the LSI ex500 or an apparatus including the LSI ex500. Forexample, when the driving frequency is set lower, the voltage to beapplied to the LSI ex500 or the apparatus including the LSI ex500 isprobably set to a voltage lower than in the case where the drivingfrequency is set higher.

Furthermore, when the processing amount for decoding is larger, thedriving frequency may be set higher, and when the processing amount fordecoding is smaller, the driving frequency may be set lower as themethod for setting the driving frequency. Thus, the setting method isnot limited to the ones described above. For example, when theprocessing amount for decoding video data in conformity with MPEG-4 AVCis larger than the processing amount for decoding video data generatedby the moving picture coding method and the moving picture codingapparatus described in each of embodiments, the driving frequency isprobably set in reverse order to the setting described above.

Furthermore, the method for setting the driving frequency is not limitedto the method for setting the driving frequency lower. For example, whenthe identification information indicates that the video data isgenerated by the moving picture coding method and the moving picturecoding apparatus described in each of embodiments, the voltage to beapplied to the LSI ex500 or the apparatus including the LSI ex500 isprobably set higher. When the identification information indicates thatthe video data conforms to the conventional standard, such as MPEG-2,MPEG-4 AVC, and VC-1, the voltage to be applied to the LSI ex500 or theapparatus including the LSI ex500 is probably set lower. As anotherexample, when the identification information indicates that the videodata is generated by the moving picture coding method and the movingpicture coding apparatus described in each of embodiments, the drivingof the CPU ex502 does not probably have to be suspended. When theidentification information indicates that the video data conforms to theconventional standard, such as MPEG-2, MPEG-4 AVC, and VC-1, the drivingof the CPU ex502 is probably suspended at a given time because the CPUex502 has extra processing capacity. Even when the identificationinformation indicates that the video data is generated by the movingpicture coding method and the moving picture coding apparatus describedin each of embodiments, when the CPU ex502 has extra processingcapacity, the driving of the CPU ex502 is probably suspended at a giventime. In such a case, the suspending time is probably set shorter thanthat in the case where when the identification information indicatesthat the video data conforms to the conventional standard, such asMPEG-2, MPEG-4 AVC, and VC-1.

Accordingly, the power conservation effect can be increased by switchingbetween the driving frequencies in accordance with the standard to whichthe video data conforms. Furthermore, when the LSI ex500 or theapparatus including the LSI ex500 is driven using a battery, the batterylife can be extended with the power conservation effect.

Embodiment 14

There are cases where a plurality of video data that conforms todifferent standards, is provided to the devices and systems, such as atelevision and a cellular phone. In order to enable decoding theplurality of video data that conforms to the different standards, thesignal processing unit ex507 of the LSI ex500 needs to conform to thedifferent standards. However, the problems of increase in the scale ofthe circuit of the LSI ex500 and increase in the cost arise with theindividual use of the signal processing units ex507 that conform to therespective standards.

In order to solve the problem, what is conceived is a configuration inwhich the decoding processing unit for implementing the moving picturedecoding method described in each of embodiments and the decodingprocessing unit that conforms to the conventional standard, such asMPEG-2, MPEG-4 AVC, and VC-1 are partly shared. Ex900 in (a) of FIG. 41shows an example of the configuration. For example, the moving picturedecoding method described in each of embodiments and the moving picturedecoding method that conforms to MPEG-4 AVC have, partly in common, thedetails of processing, such as entropy coding, inverse quantization,deblocking filtering, and motion compensated prediction. The details ofprocessing to be shared probably include use of a decoding processingunit ex902 that conforms to MPEG-4 AVC. In contrast, a dedicateddecoding processing unit ex901 is probably used for other processingunique to an aspect of the present invention. Since the aspect of thepresent invention is characterized by inverse quantization inparticular, for example, the dedicated decoding processing unit ex901 isused for inverse quantization. Otherwise, the decoding processing unitis probably shared for one of the entropy decoding, deblockingfiltering, and motion compensation, or all of the processing. Thedecoding processing unit for implementing the moving picture decodingmethod described in each of embodiments may be shared for the processingto be shared, and a dedicated decoding processing unit may be used forprocessing unique to that of MPEG-4 AVC.

Furthermore, ex1000 in (b) of FIG. 41 shows another example in thatprocessing is partly shared. This example uses a configuration includinga dedicated decoding processing unit ex1001 that supports the processingunique to an aspect of the present invention, a dedicated decodingprocessing unit ex1002 that supports the processing unique to anotherconventional standard, and a decoding processing unit ex1003 thatsupports processing to be shared between the moving picture decodingmethod according to the aspect of the present invention and theconventional moving picture decoding method. Here, the dedicateddecoding processing units ex1001 and ex1002 are not necessarilyspecialized for the processing according to the aspect of the presentinvention and the processing of the conventional standard, respectively,and may be the ones capable of implementing general processing.Furthermore, the configuration of the present embodiment can beimplemented by the LSI ex500.

As such, reducing the scale of the circuit of an LSI and reducing thecost are possible by sharing the decoding processing unit for theprocessing to be shared between the moving picture decoding methodaccording to the aspect of the present invention and the moving picturedecoding method in conformity with the conventional standard.

The present invention can help increase coding efficiency, and isapplicable to a variety of uses such as storage, transmission, andcommunication. For example, the present invention can be applicable toand is highly useful for high-resolution information display devices andimaging devices such as a television receiver, a digital video recorder,a car navigation system, a mobile phone, a digital camera.

REFERENCE SIGNS LIST

-   -   100, 1300, 2200 Image coding apparatus    -   101, 1301 Frame memory    -   102, 1302, 2209 Motion estimation unit    -   103, 908, 1303, 1708, 2206, 2305 Reference picture memory    -   104, 1304 Coding control unit    -   105, 906, 1305, 1706, 2212, 2309 Boundary filtering unit    -   106, 1306, 2201 Difference unit    -   107, 1307 Transformation unit    -   108, 1308 Quantization unit    -   109, 1309, 2211 Variable length coding unit    -   110, 902, 1310, 1702 Inverse quantization unit    -   111, 903, 1311, 1703 Inverse transformation unit    -   112, 904, 1312, 1704, 2204, 2303 Addition unit    -   900, 1700, 2300 Image decoding apparatus    -   901, 1701, 2301 Variable length decoding unit    -   905, 1705 Decoding control unit    -   907, 1707 Motion compensation unit    -   1313 Filter information description unit    -   1709 Filter information decoding unit    -   2210, 2308 Switch    -   2202 Transformation and quantization unit    -   2203, 2302 Inverse quantization and inverse transformation unit    -   2205, 2304 Coding distortion removal unit    -   2207, 2306 Intra predicting unit    -   2208, 2307 Motion compensation predicting unit

The invention claimed is:
 1. An image coding method for coding an inputimage per block to generate a coded image signal, the image codingmethod comprising: performing prediction process on prediction unitseach of which is an area obtained by partitioning a target block, togenerate a prediction image of the target block; performing boundaryfiltering process on a boundary of each of the prediction units in thegenerated prediction image, when the boundary is located within atransform unit, the transform unit being an area obtained bypartitioning the target block and being a processing unit for frequencytransform; calculating a difference between the prediction image afterthe filtering process and the input image, to generate a differenceimage of the target block; and performing frequency transform on thedifference image for each of the transform units to generate the codedimage signal.
 2. The image coding method according to claim 1, whereinno filtering process is performed on the boundary of the transform unit.3. The image coding method according to claim 1, further comprisingdescribing, in the coded image signal, filter information indicating amethod for the boundary filtering process.
 4. The image coding methodaccording to claim 1, wherein the performing of prediction processincludes: performing motion estimation for each of the prediction unitsto obtain a motion vector for the prediction unit; and performing motioncompensation for each of the prediction units based on the obtainedmotion vector, to generate a motion compensated image of the targetblock as the prediction image.
 5. The image coding method according toclaim 1, wherein the performing of prediction process includes intraprediction for each of the prediction units based on a reconstructedimage of a coded block, to generate an intra prediction image of thetarget block as the prediction image.
 6. The image coding methodaccording to claim 1, wherein the performing of prediction processincludes: performing motion estimation for each of the prediction unitsto obtain a motion vector for the prediction unit; performing motioncompensation for each of the prediction units based on the obtainedmotion vector, to generate a motion compensated image of the targetblock; performing intra prediction for each of the prediction unitsbased on a reconstructed image of a coded block, to generate an intraprediction image of the target block; and comparing the motioncompensated image with the intra prediction image to select one of themotion compensated image and the intra prediction image as theprediction image.
 7. An image coding apparatus for coding an input imageper block to generate a coded image signal, the image coding apparatuscomprising: a processor; and a non-transitory memory having a computerprogram recorded thereon, the computer program when executed by theprocessor is operable to cause the image coding apparatus to performoperations comprising: performing prediction process on prediction unitseach of which is an area obtained by partitioning a target block, togenerate a prediction image of the target block; perform filteringprocess on a boundary of each of the prediction units in the generatedprediction image, when the boundary is located within a transform unit,the transform unit being an area obtained by partitioning the targetblock and being a processing unit for frequency transform; calculating adifference between the prediction image after the filtering process andthe input image, to generate a difference image of the target block; andperforming frequency transform on the difference image for each of thetransform units to generate the coded image signal.
 8. An image decodingmethod for decoding a coded signal obtained by coding an input image perblock, the image decoding method comprising: receiving the coded imagesignal; performing prediction process on prediction units each of whichis an area obtained by partitioning a target block, to generate aprediction image of the target block; performing boundary filteringprocess on a boundary of each of the prediction units in the generatedprediction image, when the boundary is located within a transform unit,the transform unit being an area obtained by partitioning the targetblock and being a processing unit for frequency transform; calculating adifference between the prediction image after the filtering process andthe input image, to generate a difference image of the target block; andperforming an inverse frequency transform on the difference image foreach of the transform units to generate a decoded image signal.
 9. Theimage decoding method according to claim 8, wherein no filtering processis performed on the boundary of the transform unit.
 10. The imagedecoding method according to claim 8, further comprising describing, inthe coded image signal, filter information indicating a method for theboundary filtering process.
 11. The image decoding method according toclaim 8, wherein the performing of prediction process includes:performing motion estimation for each of the prediction units to obtaina motion vector for the prediction unit; and performing motioncompensation for each of the prediction units based on the obtainedmotion vector, to generate a motion compensated image of the targetblock as the prediction image.
 12. The image decoding method accordingto claim 8, wherein the performing of prediction process includes intraprediction for each of the prediction units based on a reconstructedimage of a coded block, to generate an intra prediction image of thetarget block as the prediction image.
 13. The image decoding methodaccording to claim 8, wherein the performing of prediction processincludes: performing motion estimation for each of the prediction unitsto obtain a motion vector for the prediction unit; performing motioncompensation for each of the prediction units based on the obtainedmotion vector, to generate a motion compensated image of the targetblock; performing intra prediction for each of the prediction unitsbased on a reconstructed image of a coded block, to generate an intraprediction image of the target block; and comparing the motioncompensated image with the intra prediction image to select one of themotion compensated image and the intra prediction image as theprediction image.
 14. An image decoding apparatus for decoding a codedimage signal obtained by coding an input image per block, the imagedecoding apparatus comprising: a processor; and a non-transitory memoryhaving a computer program recorded thereon, the computer program whenexecuted by the processor is operable to cause the image decodingapparatus to perform operations comprising: receiving the coded imagesignal; performing prediction process on prediction units each of whichis an area obtained by partitioning a target block, to generate aprediction image of the target block; performing boundary filteringprocess on a boundary of each of the prediction units in the generatedprediction image, when the boundary is located within a transform unit,the transform unit being an area obtained by partitioning the targetblock and being a processing unit for frequency transform; calculating adifference between the prediction image after the filtering process andthe input image, to generate a difference image of the target block; andperforming an inverse frequency transform on the difference image foreach of the transform units to generate a decoded image signal.